:W&BᾉۗºM 4xyaxc7dª; 9 /0.29.0+7872fd9d9f86ca5c50617e19cb168122b7f959480.70.0$ –ù‚" zepao172ofiv 4xyaxc7dŠÕ 4xyaxc7djuju11-uc-san-diegoSPAtop"˜ _wandb‚{}  batch_size‚256  dataset_limit‚0.2  dropout‚0.03581317281051733 epochs‚10  hidden_dim‚64 ' learning_rate‚0.0002149866136400636  num_attention_heads‚8  num_branch_encoder_layers‚3  num_embedding_layers‚6  num_encoder_layers‚6Bjjtttt_vanilla_4xyaxc7db88oquqqwj)jj-spatop-sweepagent-tttt-vanilla-1-k2mn8Š îþâÔð«ë§š3 (H¨¸hp"3.12.10*0.29.0j linux-x86_64Ëô7 4xyaxc7d&  _wandb.runtime‚0  _runtime‚0¡ÈŽgjÒä -Linux-6.8.0-136-generic-x86_64-with-glibc2.36CPython 3.12.10 îþâÔÐðÙš*-of*=/data/spatop/options_files/sweep_tttt_jj/config_4xyaxc7d.json*-l* /scratch/logs*-n*jjtttt_vanilla_4xyaxc7d2-m spanet.trainRjacksonjulian2014@gmail.comZ /scratch/logsb)jj-spatop-sweepagent-tttt-vanilla-1-k2mn8r/opt/conda/bin/pythonº 6xdc13iuzay6azd4odfadiqnio4frbt0Ó DÉÒÀ€ˆ€’NVIDIA A100-SXM4-80GB˜¢ /€€€¥È.€€ÈÝ̪€àãÓ½ÂS NVIDIA A100-SXM4-80GB€€€€À€6"Ampere*(GPU-78f24734-e3a1-59a7-ce1b-bb4283f77c97Ê13.2º 6xdc13iuzay6azd4odfadiqnio4frbt0µª½Ç2  requirements.txt=N¹ÍW 4xyaxc7dZF (H¨¸ (H¨¸ (hpð"3.12.10*0.29.0j linux-x86_64÷¤KY 4xyaxc7dZH (H¨¸ (H¨¸ (8hpð"3.12.10*0.29.0j linux-x86_64Ó߆‡*  4xyaxc7db trainer/global_step2.§N\  4xyaxc7dZK (H¨¸ (H¨¸(8hpð"3.12.10*0.29.0j linux-x86_64;Þ“/  4xyaxc7db*"trainer/global_step2ГüŸK  4xyaxc7dj: ïþâÔ°úÍÑ*LOCAL_RANK: 0 - CUDA_VISIBLE_DEVICES: [0] ‚”s“  4xyaxc7dj  ïþâÔèŸÖ®î â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┳â”â”â”â”â”â”â”┓ ┃ ┃ Name ┃ Type ┃ Params ┃ Mode ┃ FLOPs ┃ ┡â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”╇â”â”â”â”â”â”â”┩ │ 0 │ embedding │ MultiInputVectorEmbed… │ 220 K │ train │ 0 │ │ 1 │ encoder │ JetEncoder │ 498 K │ train │ 0 │ │ 2 │ branch_decoders │ ModuleList │ 9.6 M │ train │ 0 │ │ 3 │ regression_decoder │ RegressionDecoder │ 0 │ train │ 0 │ │ 4 │ classification_decoder │ ClassificationDecoder │ 0 │ train │ 0 │ │ │ other params │ n/a │ 48 │ n/a │ n/a │ └───┴────────────────────────┴────────────────────────┴────────┴───────┴───────┘ û[Œ\ 4xyaxc7djÊ ïþâÔˆÝɯ·Trainable params: 10.3 M Non-trainable params: 86 Total params: 10.3 M Total estimated model params size (MB): 41.196 Modules in train mode: 4413 Modules in eval mode: 0 Total FLOPs: 0 @öZq 4xyaxc7dj` ïþâÔˆ›ß¯Pwandb: WARNING Config item 'hidden_dim' was locked by 'sweep' (ignored update). jÊk{ 4xyaxc7djj ïþâÔð‡è¯Zwandb: WARNING Config item 'num_embedding_layers' was locked by 'sweep' (ignored update). ÆÚ]ƒy 4xyaxc7djh ïþâÔ¨þò¯Xwandb: WARNING Config item 'num_encoder_layers' was locked by 'sweep' (ignored update). qØ{‡€ 4xyaxc7djo ïþâÔЗ‚°_wandb: WARNING Config item 'num_branch_encoder_layers' was locked by 'sweep' (ignored update). É]D…z 4xyaxc7dji ïþâÔÈ¥š°Ywandb: WARNING Config item 'num_attention_heads' was locked by 'sweep' (ignored update). YbZ6t 4xyaxc7djc ïþâÔ诡°Swandb: WARNING Config item 'dataset_limit' was locked by 'sweep' (ignored update). ërèkq 4xyaxc7dj` ïþâÔ¸¡«°Pwandb: WARNING Config item 'batch_size' was locked by 'sweep' (ignored update). 1]t 4xyaxc7djc ïþâÔ¨»²°Swandb: WARNING Config item 'learning_rate' was locked by 'sweep' (ignored update). ϳˆŽn 4xyaxc7dj] ïþâÔà±½°Mwandb: WARNING Config item 'dropout' was locked by 'sweep' (ignored update). sŠøgm 4xyaxc7dj\ ïþâÔðäÆ°Lwandb: WARNING Config item 'epochs' was locked by 'sweep' (ignored update). c0F†_ 4xyaxc7d*Í  transformer_dim‚64  transformer_dim_scale‚2.0  initial_embedding_dim‚64  position_embedding_dim‚64 ! num_branch_embedding_layers‚3  num_jet_embedding_layers‚0  num_jet_encoder_layers‚2  num_detector_layers‚2  num_regression_layers‚3  num_classification_layers‚3  split_symmetric_attention‚1 ! transformer_activation‚"gelu"  skip_connections‚1 ( "initial_embedding_skip_connections‚1  linear_block_type‚"GRU"  transformer_type‚"Gated"  linear_activation‚"gelu"  normalization‚ "LayerNorm"  masking‚ "Filling" linear_prelu_activation‚true  use_pairwise_interactions‚0  num_pairwise_features‚4  pairwise_embedding_dim‚8  pairwise_input_source‚""  use_moe‚false  num_experts‚8  num_experts_per_tok‚2  moe_loss_scale‚0.01 F event_info_file‚2"/data/spatop/event_files/tttt/tttt_hadronic.yaml" = training_file‚+"/data/spatop/tttt_15M/h5/tttt_training.h5"  validation_file‚""  testing_file‚""  normalize_features‚1  limit_to_num_jets‚0  balance_particles‚0  balance_jets‚0 ! balance_classifications‚false  partial_events‚1  dataset_randomization‚0  train_validation_split‚0.95  limit_val_batches‚1.0  num_dataloader_workers‚0  mask_sequence_vectors‚1  combine_pair_loss‚"min"  optimizer‚"AdamW"  focal_gamma‚0.0  combinatorial_scale‚0.0 # learning_rate_warmup_epochs‚1.0  learning_rate_cycles‚1  assignment_loss_scale‚1.0  detection_loss_scale‚1.0  kl_loss_scale‚0.0  regression_loss_scale‚0.0 ! classification_loss_scale‚0.0 ! use_mass_classification‚false  mass_classes‚[]  balance_losses‚true  l2_penalty‚0.0002  gradient_clip‚10.0 num_gpu‚1  verbose_output‚false  usable_gpus‚""  trial_time‚"" $ trial_output_dir‚"./test_output"g¨y¡ë 4xyaxc7djÙ ïþâÔàƧµÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. ½Ï'N 4xyaxc7dj¼ ïþâÔ´µ«/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'val_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. Æ4Õôë 4xyaxc7djÙ òþâÔ˜›ÚæÈ/opt/conda/lib/python3.12/site-packages/pytorch_lightning/utilities/_pytree.py:21: `isinstance(treespec, LeafSpec)` is deprecated, use `isinstance(treespec, TreeSpec) and treespec.is_leaf()` instead. Fh&ïP 4xyaxc7dj¾ òþâÔ¨­îæ­/opt/conda/lib/python3.12/site-packages/pytorch_lightning/trainer/connectors/data_connector.py:434: The 'train_dataloader' does not have many workers which may be a bottleneck. Consider increasing the value of the `num_workers` argument` to `num_workers=3` in the `DataLoader` to improve performance. Ç|­í‚(:† þþâÔБÀ® gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5039825439453125) gpu.0.memoryAllocatedBytes‚ 5586878464 gpu.0.temp‚32.0 gpu.0.powerWatts‚130.245' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.049 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0.Á8$‚(:Š þþâÔÊ´µ disk.drbd2080.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17% disk./.usageGB‚243.18265533447266 disk.drbd2266.out‚0 cpu‚25.619128915636704 disk.sda.out‚ 16.8515625 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 network.recv‚41624 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.dm-0.in‚0 disk.dm-0.out‚ 0.40234375 disk.sda1.in‚0 disk.nvme0c0n1.in‚0 disk.dm-1.out‚ 16.8515625 disk.nvme0n1.out‚ 0.40234375 disk.sda3.in‚0 disk.rbd1.out‚0$ memory_percent‚85.47203540802002# proc.memory.rssMB‚ 2588.20703125 disk.sda3.out‚ 16.8515625 network.sent‚34019 disk.nvme0n1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 2380.26171875 disk.rbd1.in‚0! disk.nvme0c0n1.out‚ 0.40234375* proc.memory.percent‚15.797162055969238* disk./.usagePercent‚16.320192043334988 disk.sda.in‚0 disk.drbd2266.in‚0 disk.drbd2265.out‚0ËÊ…Bœ ‚(:’ ÿâÔÉçž® gpu.0.gpu‚32.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.558' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.111600000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ëÏ}!‚(: ÿâԧ瘱* disk./.usagePercent‚16.320195371432934 disk.sda2.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0$ disk./.usageGB‚243.1827049255371 disk.nvme0c0n1.in‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 34.6640625 cpu‚25.463353194128885 network.recv‚59707 disk.sda.in‚0 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.in‚0 network.sent‚51182 disk.rbd1.in‚0 disk.rbd1.out‚0.15625 disk.nvme0n1.out‚ 0.822265625 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 34.6640625 disk.nvme0n1.in‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0$ memory_percent‚85.49618721008301# proc.memory.rssMB‚ 2588.23046875 proc.cpu.threads‚17 disk.dm-0.out‚ 0.822265625( proc.memory.percent‚15.7973051071167" disk.nvme0c0n1.out‚ 0.822265625 disk.sda.out‚ 34.6640625 disk.dm-0.in‚0( proc.memory.availableMB‚ 2376.3046875ÍÈB 3"b/ lr-AdamW/pg2"trainer/global_step2:JPf"õå3#b/ lr-AdamW/pg1"trainer/global_step2:JP¯ˆ-$) _wandbruntime‚40  _runtime‚40J’ê9À%» % _timestamp‚1.7883954158406022e+09 _runtime‚ 40.477555004 _step‚0 & lr-AdamW/pg1‚8.713270528009195e-06 & lr-AdamW/pg2‚8.713270528009195e-06 trainer/global_step‚496ìûÂ&½ & lr-AdamW/pg1‚8.713270528009195e-06 & lr-AdamW/pg2‚8.713270528009195e-06 trainer/global_step‚49 % _timestamp‚1.7883954158406022e+09 _runtime‚ 40.477555004 _step‚0g‚hzn'bj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JPßá°n(bj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JP7§úe)ba >DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed"trainer/global_step2:JPE×Ó¤n*bj GDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current"trainer/global_step2:JPbJ˜fi+be BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated"trainer/global_step2:JP¦åt,bp MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JPú{+Ëg-bc @DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak"trainer/global_step2:JPw„ fq.bm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JPH…Ö‹p/bl IDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current"trainer/global_step2:JPgÐ@ár0bn KDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JP!óSl1bh EDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPÙA…ó`2b\ 9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries"trainer/global_step2:JPñù_3b[ 8DeviceStatsMonitor.on_train_batch_start/segment.all.peak"trainer/global_step2:JP®qÆ:x4bt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP@¿`Ál5bh EDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated"trainer/global_step2:JPQ^Ûui6be BDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current"trainer/global_step2:JP?Eúàg7bc @DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed"trainer/global_step2:JP"ÄLol8bh EDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current"trainer/global_step2:JP5ú¾÷a9b] :DeviceStatsMonitor.on_train_batch_start/num_oom_rejections"trainer/global_step2:JPmEáb:b^ ;DeviceStatsMonitor.on_train_batch_start/segment.all.current"trainer/global_step2:JP5ž?÷c;b_ bn KDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated"trainer/global_step2:JP_Ä{m?bi FDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak"trainer/global_step2:JPíœ$f@bb ?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak"trainer/global_step2:JP¨Sf’hAbd ADeviceStatsMonitor.on_train_batch_start/active.large_pool.current"trainer/global_step2:JPùmu­]BbY 6DeviceStatsMonitor.on_train_batch_start/max_split_size"trainer/global_step2:JP%és­fCbb ?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed"trainer/global_step2:JP yHqDbm JDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current"trainer/global_step2:JPþø GvEbr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JPœ ¥t`Fb\ 9DeviceStatsMonitor.on_train_batch_start/segment.all.freed"trainer/global_step2:JP-_E•gGbc @DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated"trainer/global_step2:JP!»GdeHba >DeviceStatsMonitor.on_train_batch_start/allocation.all.current"trainer/global_step2:JPà  QWIbS 0DeviceStatsMonitor.on_train_batch_start/num_ooms"trainer/global_step2:JPtâý_sJbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPªPÓcKb_ DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak"trainer/global_step2:JP›h*ápWbl IDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current"trainer/global_step2:JP&£IÞfXbb ?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak"trainer/global_step2:JP|tŠgYbc @DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current"trainer/global_step2:JP)tk&sZbo LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated"trainer/global_step2:JP§DÝ#q[bm JDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JP~æn\m\bi FDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JPQmüf]bb ?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed"trainer/global_step2:JP_ÓTs^bo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated"trainer/global_step2:JP…›*Ío_bk HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JP12n`bj GDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated"trainer/global_step2:JP£ §š^abZ 7DeviceStatsMonitor.on_train_batch_start/active.all.peak"trainer/global_step2:JP·ÿQkbbg DDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JP¢þlcbh EDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current"trainer/global_step2:JPFÝ7ªxdbt QDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JPæÁ{iebe BDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed"trainer/global_step2:JP4úÏMnfbj GDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JP¡çPG_gb[ 8DeviceStatsMonitor.on_train_batch_start/active.all.freed"trainer/global_step2:JP»¹Qohbk HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed"trainer/global_step2:JPÖògPgibc @DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak"trainer/global_step2:JPð›YJnjbj GDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JPh ƒñekba >DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak"trainer/global_step2:JPÇ…Ð\llbh EDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed"trainer/global_step2:JP!íPosmbo LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JPü¯.knbg DDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated"trainer/global_step2:JP‹Óïvobr ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JP^khpbd ADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak"trainer/global_step2:JPäLÎeqba >DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak"trainer/global_step2:JPãàþnrbj GDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPØÕkcsb_ DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 68902435342 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚986703 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 1376131443712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 1376113440768 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1005243 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22901 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 1456860450816 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1628418 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 1307019041792 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237701 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782413 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 68936930304 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 69105163264 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚652865 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22628 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚377054 BEk¼ 7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 1445050371072 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 1445236606976 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 1456836510720 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911252992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚275809 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 260159488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 trainer/global_step‚49 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 70476398080 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 1375938516494 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚ 40.47860714 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 1307036081152Þá27…'¤ÿN D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚1628418 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 1307019041792 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237701 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782413 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 68936930304 \LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 69105163264 ODDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚652865 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22628 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚377054 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 1445050371072 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 1445236606976 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 1456836510720 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911252992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚275809 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 260159488 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 trainer/global_step‚49 aQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 70476398080 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 1375938516494 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚ 40.47860714 HDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 1307036081152 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 \LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 68902435342 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 J?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚986703 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 1376131443712 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 1376113440768 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚1005243 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22901 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 1456860450816 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0áºr¥bm JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated"trainer/global_step2:JP¦Ú@le¦b` =DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak"trainer/global_step2:JP©e˜gj§be BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated"trainer/global_step2:JPˆŒ™Ùg¨bb ?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current"trainer/global_step2:JPçGaÎc©b^ ;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak"trainer/global_step2:JPªÈÝvhªbc @DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak"trainer/global_step2:JPç‚·j«be BDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current"trainer/global_step2:JPå»9µe¬b` =DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed"trainer/global_step2:JPëço­bj GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated"trainer/global_step2:JP×–Š¾_®bZ 7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries"trainer/global_step2:JPc=Ê9e¯b` =DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak"trainer/global_step2:JPå“çlj°be BDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak"trainer/global_step2:JPð¨ùm±bh EDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated"trainer/global_step2:JP¸ï4±q²bl IDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated"trainer/global_step2:JPZ«Ýýp³bk HDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current"trainer/global_step2:JP5iÊbd ADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current"trainer/global_step2:JPkeÃhËbc @DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current"trainer/global_step2:JPÀ®ý\ÌbW 4DeviceStatsMonitor.on_train_batch_end/max_split_size"trainer/global_step2:JP¤³Ú¸hÍbc @DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed"trainer/global_step2:JP]Wn\bÎb] :DeviceStatsMonitor.on_train_batch_end/active.all.allocated"trainer/global_step2:JPÖÇxÁiÏbd ADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated"trainer/global_step2:JPºSj"mÐbh EDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current"trainer/global_step2:JPU¹rÑbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JPÉ¡¡fÒba >DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed"trainer/global_step2:JP¥ÂE€iÓbd ADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed"trainer/global_step2:JPK$!¨mÔbh EDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JPØ·JyfÕba >DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed"trainer/global_step2:JPnVîžjÖbe BDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPÛïòe×b` =DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak"trainer/global_step2:JPànÃidØb_ DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated"trainer/global_step2:JP0Ö—šhÚbc @DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak"trainer/global_step2:JP Ò mÛbh EDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPa\Ú½qÜbl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JPßð<fÝba >DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current"trainer/global_step2:JP֭ΪkÞbf CDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated"trainer/global_step2:JP^Í_rßbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPø´ÝÛ]àbX 5DeviceStatsMonitor.on_train_batch_end/num_device_free"trainer/global_step2:JPÊ8gább ?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current"trainer/global_step2:JPΟiâbd ADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed"trainer/global_step2:JP·¹gÔrãbm JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated"trainer/global_step2:JP нâhäbc @DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current"trainer/global_step2:JPlŠmåbh EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed"trainer/global_step2:JP¯É–ýmæbh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak"trainer/global_step2:JPÞ’Ú^çbY 6DeviceStatsMonitor.on_train_batch_end/segment.all.peak"trainer/global_step2:JPñ¨amèbh EDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated"trainer/global_step2:JP¸¯þ…jébe BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated"trainer/global_step2:JPè&úlêbg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak"trainer/global_step2:JPÕ¯0]ëbX 5DeviceStatsMonitor.on_train_batch_end/active.all.peak"trainer/global_step2:JPÙ˜Ô dìb_ DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed"trainer/global_step2:JPö&ˆm‚bh EDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JPïBØrƒbm JDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak"trainer/global_step2:JP“ÀÏú`„b[ 8DeviceStatsMonitor.on_train_batch_end/active.all.current"trainer/global_step2:JPÎWQoh…bc @DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated"trainer/global_step2:JPóOm†bh EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JPnI’yj‡be BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated"trainer/global_step2:JP“;oˆbj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JP ÁÂŽl‰bg DDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak"trainer/global_step2:JP²œ¶­_ŠbZ 7DeviceStatsMonitor.on_train_batch_end/segment.all.freed"trainer/global_step2:JP WXÐg‹bb ?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed"trainer/global_step2:JPí•„XsŒbn KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPÈ obj GDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current"trainer/global_step2:JP‘x÷eŽb` =DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed"trainer/global_step2:JPßû‚kubp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JP‚b¾^fba >DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak"trainer/global_step2:JPÙßEf‘ba >DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak"trainer/global_step2:JPnœ`ýn’bi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JP§ÿ=6n“bi FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed"trainer/global_step2:JP|0MÌk”bf CDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current"trainer/global_step2:JPDž#o•bj GDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current"trainer/global_step2:JPÑ2 k–bf CDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JP åú&u—bp MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current"trainer/global_step2:JPüå¿&q˜bl IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated"trainer/global_step2:JPqëm™bh EDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current"trainer/global_step2:JPwëmÿdšb_ DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed"trainer/global_step2:JPÖ]iŸbd ADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current"trainer/global_step2:JPÄOYk. ) _wandbruntime‚41  _runtime‚41Å=hýa¡ˆM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚1660864 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 70330068480 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1660864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 70126486030 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1557425085440 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 1403270863582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 1332962224128 _step‚2 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 70498314752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚665059 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚280628 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 _runtime‚ 41.180650249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 1473899123200 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 1473712873984 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 71675626496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚654374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 207143936 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 1403382805504 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 70330068480 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 260159488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 1403400808448 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 1403400808448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚654376 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚384433 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 NDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1006490 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚968 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1679432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚384431 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 1473899123200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚654374 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 70291600094 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1025056 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚665582 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 70330068480 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚1660864 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 VFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 70126486030 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 1557425085440 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 1403270863582 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 1332962224128 _step‚2 WGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 70498314752 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚665059 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚280628 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 _runtime‚ 41.180650249 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 1473899123200 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 1473712873984 [KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 71675626496 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚654374 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 207143936 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 1403382805504 VFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 70330068480 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 260159488 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 1403400808448 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 1403400808448 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 PEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚654376 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚384433 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 NDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚1006490 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚968 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚1679432 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚384431 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 1473899123200 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 LADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚654374 ZJDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 70291600094 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚1025056 MBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚665582 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 E loss/SRqqt2/assignment_loss"trainer/global_step2:JP ,Ù—B«b= loss/SRqqt2/detection_loss"trainer/global_step2:JPÊ¢A¬b< loss/FRt1/assignment_loss"trainer/global_step2:JP©HÃÔA­b< loss/FRt4/assignment_loss"trainer/global_step2:JPÂH3B®b= loss/SRqqt1/detection_loss"trainer/global_step2:JPò^‘'@¯b; loss/FBt4/detection_loss"trainer/global_step2:JPV’]‰C°b> loss/SRqqt3/assignment_loss"trainer/global_step2:JP˜¦“A±b< loss/FBt4/assignment_loss"trainer/global_step2:JP\ã¨Ç@²b; loss/FRt1/detection_loss"trainer/global_step2:JPƒ‚m§B³b= loss/SRqqt3/detection_loss"trainer/global_step2:JPÈ™¦C´b> loss/SRqqt1/assignment_loss"trainer/global_step2:JP•hjp@µb; loss/FRt4/detection_loss"trainer/global_step2:JPqA¶b< loss/FBt1/assignment_loss"trainer/global_step2:JP-}’Þ@·b; loss/FBt3/detection_loss"trainer/global_step2:JP|y7Ä-¸b( epoch"trainer/global_step2:JP‡X¯µA¹b< loss/FRt2/assignment_loss"trainer/global_step2:JPì÷ûÕAºb< loss/FBt2/assignment_loss"trainer/global_step2:JPFÌ[C»b> loss/SRqqt4/assignment_loss"trainer/global_step2:JP‚°ã]@¼b; loss/FRt2/detection_loss"trainer/global_step2:JP’>..½) _wandbruntime‚41  _runtime‚41'ŠH¾ 0loss/FBt1/assignment_loss‚2.7694475650787354 -loss/FBt3/detection_loss‚4.22946892365029 epoch‚0 _runtime‚ 41.181947026 /loss/FRt2/assignment_loss‚7.354810313174599 0loss/FBt2/assignment_loss‚2.7015672624111176 0loss/SRqqt4/assignment_loss‚5.43677597225837 .loss/FRt2/detection_loss‚4.522015632767427 0loss/SRqqt4/detection_loss‚3.157499796658192 .loss/FBt1/detection_loss‚8.115772814977737 /loss/FRt3/assignment_loss‚7.266496647235959 0loss/FBt3/assignment_loss‚2.7957953277387118 /loss/FRt3/detection_loss‚3.8725115067737046 .loss/FBt2/detection_loss‚6.983447048813105 &loss/total_loss‚117.87953432121024 1loss/SRqqt2/assignment_loss‚5.486779451370239 0loss/SRqqt2/detection_loss‚3.719078441235152 _step‚3 /loss/FRt1/assignment_loss‚7.255496299266815 /loss/FRt4/assignment_loss‚7.250675791785831 0loss/SRqqt1/detection_loss‚4.597040766163876 .loss/FBt4/detection_loss‚7.448784264533416 % _timestamp‚1.7883954165451164e+09 2loss/SRqqt3/assignment_loss‚5.3385893056090445 0loss/FBt4/assignment_loss‚2.7190808938897173 .loss/FRt1/detection_loss‚4.268763739056885 0loss/SRqqt3/detection_loss‚2.330390578424427 trainer/global_step‚49 2loss/SRqqt1/assignment_loss‚5.4910095867357755 /loss/FRt4/detection_loss‚2.76823639160110839-L¿Æ 0loss/SRqqt2/detection_loss‚3.719078441235152 _step‚3 /loss/FRt1/assignment_loss‚7.255496299266815 /loss/FRt4/assignment_loss‚7.250675791785831 0loss/SRqqt1/detection_loss‚4.597040766163876 .loss/FBt4/detection_loss‚7.448784264533416 % _timestamp‚1.7883954165451164e+09 2loss/SRqqt3/assignment_loss‚5.3385893056090445 0loss/FBt4/assignment_loss‚2.7190808938897173 .loss/FRt1/detection_loss‚4.268763739056885 0loss/SRqqt3/detection_loss‚2.330390578424427 trainer/global_step‚49 2loss/SRqqt1/assignment_loss‚5.4910095867357755 /loss/FRt4/detection_loss‚2.7682363916011083 0loss/FBt1/assignment_loss‚2.7694475650787354 -loss/FBt3/detection_loss‚4.22946892365029 epoch‚0 _runtime‚ 41.181947026 /loss/FRt2/assignment_loss‚7.354810313174599 0loss/FBt2/assignment_loss‚2.7015672624111176 0loss/SRqqt4/assignment_loss‚5.43677597225837 .loss/FRt2/detection_loss‚4.522015632767427 0loss/SRqqt4/detection_loss‚3.157499796658192 .loss/FBt1/detection_loss‚8.115772814977737 /loss/FRt3/assignment_loss‚7.266496647235959 0loss/FBt3/assignment_loss‚2.7957953277387118 /loss/FRt3/detection_loss‚3.8725115067737046 .loss/FBt2/detection_loss‚6.983447048813105 &loss/total_loss‚117.87953432121024 1loss/SRqqt2/assignment_loss‚5.486779451370239Èù-:’À‚(:‡ œÿâÔŽ•Ô® gpu.0.gpu‚21.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚129.179' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.8358 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŸÐ& #Á‚(:˜ œÿâÔ†½¢± disk.nvme0n1.in‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0 network.recv‚78054 disk.sda2.out‚0 disk.drbd2266.out‚0) proc.memory.availableMB‚ 2374.98828125) proc.memory.percent‚15.79749584197998* disk./.usagePercent‚16.320199723561014" disk.nvme0n1.out‚ 1.00830078125 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 cpu‚25.374670988583233 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 52.7578125 disk.sda2.in‚0 disk.dm-1.out‚ 52.7578125 disk.drbd2080.in‚0 disk.dm-0.in‚0$ memory_percent‚85.50422191619873 disk.sda.in‚0 disk.sda.out‚ 52.7578125 disk.sda3.in‚0 disk.rbd1.out‚0.15625# proc.memory.rssMB‚ 2588.26171875 proc.cpu.threads‚17 disk.nvme0c0n1.in‚0% disk./.usageGB‚243.18276977539062 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.dm-0.out‚ 1.00830078125$ disk.nvme0c0n1.out‚ 1.00830078125 network.sent‚64640 disk.drbd2080.out‚0>Áa‚(:’ «ÿâÔ¹® gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚124.287' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.857400000000002 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A‡É´$Â(:™ «ÿâÔ¹÷ÿ°* disk./.usagePercent‚16.320200235576085 disk.dm-1.in‚0 disk.dm-1.out‚ 72.43359375 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.dm-0.in‚0# proc.memory.rssMB‚ 2588.27734375% disk./.usageGB‚243.18277740478516! disk.nvme0c0n1.out‚ 2.10546875 disk.sda.in‚0 disk.sda2.in‚0 disk.drbd2265.out‚0* proc.memory.percent‚15.797591209411621 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 72.43359375 network.sent‚134329 network.recv‚111889 disk.dm-0.out‚ 2.10546875) proc.memory.availableMB‚ 2372.63671875 disk.sda.out‚ 72.43359375 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚0$ memory_percent‚85.51857471466064 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.nvme0n1.out‚ 2.10546875 cpu‚25.32044247063994 proc.cpu.threads‚17 disk.sda3.in‚0 disk.nvme0n1.in‚0 disk.rbd1.out‚ 0.1884765625>-›’Ä‚(:‡ ºÿâÔä…® gpu.0.gpu‚22.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚131.601' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.3202 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–ÒÈÅ‚(: ºÿâÔÏ·Ó± disk.sda2.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.out‚ 2.3203125 disk.nvme0n1.out‚ 2.3203125 disk.rbd1.in‚0) proc.memory.availableMB‚ 2374.28515625 cpu‚25.27951790784011 disk.drbd2080.out‚0# proc.memory.rssMB‚ 2588.27734375 network.sent‚141173 disk.sda.in‚0 disk.sda.out‚ 86.328125 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0$ memory_percent‚85.50851345062256 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 86.328125 disk.rbd1.out‚ 0.2744140625 disk.nvme0c0n1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-0.out‚ 2.3203125 disk.nvme0n1.in‚0* proc.memory.percent‚15.797591209411621* disk./.usagePercent‚16.320198955538416% disk./.usageGB‚243.18275833129883 disk.dm-1.out‚ 86.328125 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚117251µï.Æ) _wandbruntime‚77  _runtime‚77Ý^ŸüÁÇ» _step‚4 & lr-AdamW/pg1‚1.760436290352878e-05 & lr-AdamW/pg2‚1.760436290352878e-05 trainer/global_step‚99 % _timestamp‚1.7883954530731106e+09 _runtime‚ 77.709975836^Ö‘ŠÅÈ¿ % _timestamp‚1.7883954530731106e+09 _runtime‚ 77.709975836 _step‚4 & lr-AdamW/pg1‚1.760436290352878e-05 & lr-AdamW/pg2‚1.760436290352878e-05 trainer/global_step‚99Q1æ/.É) _wandbruntime‚77  _runtime‚77‡:«óÔ#ÊŒO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209254400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 2604195197952 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 2878189930496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚99 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3254070 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚954 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 141129977856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 2878376166400 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1997945 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 2604178158592 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1274665 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚556 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2878189930496 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 138776485888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 % _timestamp‚1.7883954530738754e+09 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 2742569451702 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 2742387298486 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233194496 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 141339232256 _step‚5 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 138776485888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 _runtime‚ 77.710964281 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1274667 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 2878376166400 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚745906 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 2878189930496 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 138776485888 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 % _timestamp‚1.7883954530738754e+09 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 2742569451702 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 2742387298486 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233194496 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 141339232256 _step‚5 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 138776485888 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 _runtime‚ 77.710964281 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1274667 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 2878376166400 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚745906 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 2604195197952 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 2878189930496 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚99 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚3254070 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚954 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 141129977856 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 2878376166400 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚1997945 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 2604178158592 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1274665 { .Ì) _wandbruntime‚78  _runtime‚78`Äò†œ&Í–M XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 140169637376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 3073944831488 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 2907038682624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚555195 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1287324 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 140169637376 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 140001391104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 2931396870144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 NDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3305084 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 78.411318166 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚554667 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 2769901798790 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 2906852433408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2017758 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3286516 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 2766869045248 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1308478 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚753281 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 2931420810240 trainer/global_step‚99 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3286516 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 2766869045248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 260159488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1287326 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 2907038682624 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7883954537739532e+09 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3305084 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 140001391104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 2766851042304 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚954 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 235278336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1307948 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 2769719645366 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1287326 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 2630121340928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 211338240 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 139598304438 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416¶Â{x &ΚM FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1287326 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 2907038682624 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7883954537739532e+09 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚3305084 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 140001391104 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 2766851042304 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚954 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 235278336 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1307948 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 2769719645366 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚1287326 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 2630121340928 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 211338240 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 139598304438 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 140169637376 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 3073944831488 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 2907038682624 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 TIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚555195 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1287324 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 140169637376 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 140001391104 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 2931396870144 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 NDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚3305084 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚ 78.411318166 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚554667 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 2769901798790 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 2906852433408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚2017758 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚3286516 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 2766869045248 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1308478 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚753281 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 2931420810240 trainer/global_step‚99 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚3286516 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 2766869045248 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 260159488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199‚u°6.Ï) _wandbruntime‚78  _runtime‚78ÝB{HР1loss/SRqqt2/assignment_loss‚5.476200184923537 .loss/FRt2/assignment_loss‚7.43256089952257 1loss/SRqqt4/detection_loss‚3.1039196837406893 _runtime‚ 78.412629791 trainer/global_step‚99 /loss/FRt4/detection_loss‚3.3832765843432684 /loss/FBt2/detection_loss‚11.967392082725253 epoch‚0 /loss/FRt4/assignment_loss‚7.278110870948205 0loss/FBt2/assignment_loss‚2.6772869655064175 0loss/FBt4/assignment_loss‚2.7154715186671208 .loss/FRt2/detection_loss‚3.796109211444855 0loss/SRqqt3/detection_loss‚2.183985175066431 1loss/SRqqt4/assignment_loss‚5.495526561370263 .loss/FRt3/detection_loss‚3.389889648805062 /loss/FBt1/detection_loss‚13.469618008686947 % _timestamp‚1.7883954537758648e+09 _step‚7 1loss/SRqqt1/assignment_loss‚5.463134288787842 -loss/FRt1/detection_loss‚5.63649476136821 -loss/FBt4/detection_loss‚9.08719801902771 %loss/total_loss‚127.9892733327001 0loss/FRt1/assignment_loss‚7.2467509508132935 /loss/FRt3/assignment_loss‚7.216800093650818 /loss/FBt1/assignment_loss‚2.769315554545476 0loss/FBt3/assignment_loss‚2.7423888652221016 0loss/SRqqt1/detection_loss‚3.476619922866424 1loss/SRqqt3/assignment_loss‚5.297339117023307 1loss/SRqqt2/detection_loss‚3.4383619072589466 /loss/FBt3/detection_loss‚3.2455224563853573%pÒÌ”ÑÆ /loss/FRt4/assignment_loss‚7.278110870948205 0loss/FBt2/assignment_loss‚2.6772869655064175 0loss/FBt4/assignment_loss‚2.7154715186671208 .loss/FRt2/detection_loss‚3.796109211444855 0loss/SRqqt3/detection_loss‚2.183985175066431 1loss/SRqqt4/assignment_loss‚5.495526561370263 .loss/FRt3/detection_loss‚3.389889648805062 /loss/FBt1/detection_loss‚13.469618008686947 %`à‰¸ _timestamp‚1.7883954537758648e+09 _step‚7 1loss/SRqqt1/assignment_loss‚5.463134288787842 -loss/FRt1/detection_loss‚5.63649476136821 -loss/FBt4/detection_loss‚9.08719801902771 %loss/total_loss‚127.9892733327001 0loss/FRt1/assignment_loss‚7.2467509508132935 /loss/FRt3/assignment_loss‚7.216800093650818 /loss/FBt1/assignment_loss‚2.769315554545476 0loss/FBt3/assignment_loss‚2.7423888652221016 0loss/SRqqt1/detection_loss‚3.476619922866424 1loss/SRqqt3/assignment_loss‚5.297339117023307 1loss/SRqqt2/detection_loss‚3.4383619072589466 /loss/FBt3/detection_loss‚3.2455224563853573 1loss/SRqqt2/assignment_loss‚5.476200184923537 .loss/FRt2/assignment_loss‚7.43256089952257 1loss/SRqqt4/detection_loss‚3.1039196837406893 _runtime‚ 78.412629791 trainer/global_step‚99 /loss/FRt4/detection_loss‚3.3832765843432684 /loss/FBt2/detection_loss‚11.967392082725253 epoch‚09•('’Ò‚(:‡ ÉÿâÔÙÙ“® gpu.0.gpu‚21.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚125.106' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.0212 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£YE!Ó‚(:– ÉÿâÔóúÙ± disk.dm-0.in‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda3.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚243.1828384399414 disk.rbd1.out‚ 0.4306640625! disk.nvme0c0n1.out‚ 2.87109375* proc.memory.percent‚15.797662734985352 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚0 disk.nvme0n1.out‚ 2.87109375 disk.sda.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0$ memory_percent‚85.50915718078613 disk.dm-1.in‚0 disk.dm-1.out‚ 106.2109375 disk.dm-0.out‚ 2.87109375 disk.sda1.out‚0 disk.sda3.out‚ 106.2109375 disk.sda.out‚ 106.2109375 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2080.out‚0 network.sent‚148249 disk.drbd2266.out‚0( proc.memory.availableMB‚ 2374.1796875" proc.memory.rssMB‚ 2588.2890625 network.recv‚122570 disk.nvme0n1.in‚0 cpu‚25.24954451939952* disk./.usagePercent‚16.320204331696633 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0Õº»Ô‚(:’ ØÿâÔñôª® gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0V­õ"Õ‚(:— ØÿâÔª·ò± disk.sda.out‚ 127.3046875 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0" proc.memory.rssMB‚ 2588.2890625 disk.drbd2080.in‚0 disk.dm-0.out‚ 3.02734375 disk.sda1.in‚0 disk.nvme0c0n1.in‚0 cpu‚25.226434178029546 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 0.4306640625) proc.memory.availableMB‚ 2374.13671875 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 127.3046875* proc.memory.percent‚15.797662734985352 disk.drbd2266.out‚0 network.sent‚198012) disk./.usagePercent‚16.32020279565143 disk.sda.in‚0 disk.drbd1716.out‚0! disk.nvme0c0n1.out‚ 3.02734375 disk.nvme0n1.in‚0 network.recv‚128262$ disk./.usageGB‚243.1828155517578 disk.sda3.in‚0 disk.nvme0n1.out‚ 3.02734375 disk.sda2.in‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 127.3046875 disk.rbd0.in‚0 disk.dm-0.in‚0 disk.sda1.out‚0$ memory_percent‚85.50941944122314?lÁ‡0Ö+ _wandbruntime‚115  _runtime‚115YÍOêÅ׿ ' lr-AdamW/pg1‚2.6495455279048368e-05 ' lr-AdamW/pg2‚2.6495455279048368e-05 trainer/global_step‚149 % _timestamp‚1.7883954903709285e+09 _runtime‚ 115.007764704 _step‚8‚9\¨ÉØà _step‚8 ' lr-AdamW/pg1‚2.6495455279048368e-05 ' lr-AdamW/pg2‚2.6495455279048368e-05 trainer/global_step‚149 % _timestamp‚1.7883954903709285e+09 _runtime‚ 115.007764704ï õ¿0Ù+ _wandbruntime‚115  _runtime‚115í –'ÚO O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4311329489920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 4103049914368 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 207846072158 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2972107 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 % _timestamp‚1.7883954903716323e+09 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 207157248 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 4311329489920 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 4311515725824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 4103067917312 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚826049 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4560105204224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 4311515725824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 208279575552 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1114756 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚967 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 4103067917312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1907615 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚826579 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 231097344 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 3901337275392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 4109200386910 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2972107 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1907617 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1907617 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 208279575552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 _step‚9 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2990647 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 207680958302 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2990647 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4560336301568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚ 115.008676475 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4348155273216 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 4103049914368 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 208447808512 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 4348131333120 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 4109018233694 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 260159488 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3901354314752 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 212181028352 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1907615*ª—š'Û”O ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚1907615 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 4311329489920 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 4103049914368 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 207846072158 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚2972107 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 % _timestamp‚1.7883954903716323e+09 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 207157248 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 4311329489920 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 4311515725824 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 4103067917312 RGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚826049 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 4560105204224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 4311515725824 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 208279575552 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1114756 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚967 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 4103067917312 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚1907615 VKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚826579 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 231097344 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 3901337275392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 4109200386910 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚2972107 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚1907617 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚1907617 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 208279575552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 _step‚9 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚2990647 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 207680958302 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚2990647 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 4560336301568 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚ 115.008676475 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 4348155273216 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 4103049914368 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 208447808512 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 4348131333120 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 4109018233694 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 260159488 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 3901354314752 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 212181028352 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _-¨0Ü+ _wandbruntime‚115  _runtime‚115aR*-Ý›M KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _step‚10 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚1920274 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 209840960000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 260159488 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 4340178242048 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚831653 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚967 _runtime‚ 115.706808986 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 209235236910 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2991894 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 207143936 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚554 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4377044281344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4912168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4590437334016 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3927280457728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 209672713728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚556 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 213393052672 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1122131 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4590668418048 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 4136532733998 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 4130337282048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3010460 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 209672713728 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2991894 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 213600196608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1122133 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4377068221440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚149 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4930736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231084032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4912168 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1920276 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3010460 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1920274 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 4136350580574 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 4130337282048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚988 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3927297497088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 % _timestamp‚1.7883954910693839e+09 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1954340 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1953784 ;4DeviceStatsMonitor.on ›KSt_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 N6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 4130319279104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 209840960000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 260159488 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 4340178242048 PEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚831653 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚967 _runtime‚ 115.706808986 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 209235236910 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚2991894 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 207143936 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚554 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 4377044281344 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚4912168 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 4590437334016 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 3927280457728 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 209672713728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚556 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 213393052672 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1122131 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 4590668418048 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 4136532733998 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 4130337282048 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚3010460 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 209672713728 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚2991894 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 213600196608 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1122133 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 4377068221440 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚149 J>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚4930736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231084032 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚4912168 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚1920276 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚3010460 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚1920274 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 4136350580574 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 4130337282048 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚988 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 3927297497088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 % _timestamp‚1.7883954910693839e+09 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚1954340 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚1953784 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 N6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 4130319279104 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 277950898176 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1103025 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1102510 trainer/global_step‚199 % _timestamp‚1.7883955277334578e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3964809 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5466536154112 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198768640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 5475831322118 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 _runtime‚ 152.370551245 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚989 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3983349 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 5793778744320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 5744655285248 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 5475649168902 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 277950898176 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2540567 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1483606 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5198496392192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3983349 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 5744469049344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1483604 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 5793802684416 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3964809 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 5466536154112 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6523916 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5744469049344 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 5744655285248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2586114 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6076812139008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 5466518151168 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5198513431552 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 _step‚13 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2540567 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 277152776710 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6505374 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 5466518151168 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2586631 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6076589430272 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 277317890566 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2540565 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 278119131136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222708736 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2540565 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0*æ8ûì í—O UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 _runtime‚ 152.370551245 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚989 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚3983349 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 5793778744320 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 5744655285248 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 5475649168902 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 277950898176 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚2540567 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1483606 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 5198496392192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚3983349 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 5744469049344 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1483604 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 5793802684416 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚3964809 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 5466536154112 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚6523916 HDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 5744469049344 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 5744655285248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚2586114 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 6076812139008 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 5466518151168 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 5198513431552 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 _step‚13 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚2540567 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 277152776710 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚6505374 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 5466518151168 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚2586631 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 6076589430272 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 277317890566 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚2540565 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 278119131136 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 222708736 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚2540565 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 283009454592 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 282810685952 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 277950898176 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1103025 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1102510 trainer/global_step‚199 % _timestamp‚1.7883955277334578e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚3964809 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 5466536154112 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198768640 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 5475831322118 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 Ñ@»G0î+ _wandbruntime‚153  _runtime‚153…ö­¢&ïœM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 5773317801472 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6537820 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2553226 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5502981515782 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788395528437323e+09 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 6107152130560 trainer/global_step‚199 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4003162 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚495 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2599581 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1490981 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 5224456613888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 284227256832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚14 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2599084 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 5822715632640 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 279344036352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1108598 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚497 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚989 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6556388 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2553226 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5503163669206 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2553224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233181184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3984596 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6537820 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6556388 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 5773317801472 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 5773131552256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 278541941254 NDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0Sã/¦&ð M M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚2599581 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1490981 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 5224456613888 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 284227256832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 _step‚14 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚2599084 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 5822715632640 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 279344036352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1108598 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚497 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚989 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚6556388 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚2553226 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 UCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚ 5503163669206 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚2553224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233181184 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚3984596 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚6537820 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚6556388 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 5773317801472 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 5773131552256 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 278541941254 NDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 5773317801472 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚6537820 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚2553226 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 5502981515782 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788395528437323e+09 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 6107152130560 trainer/global_step‚199 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4003162 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚495 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160ˆª^0ñ+ _wandbruntime‚153  _runtime‚153k¿PòÊ 1loss/SRqqt1/assignment_loss‚5.440170274840461 1loss/SRqqt3/assignment_loss‚5.302434369495937 /loss/FBt1/detection_loss‚11.202747503916422 .loss/FBt2/detection_loss‚6.563832863569259 epoch‚0 /loss/FRt4/assignment_loss‚7.421305460088393 1loss/SRqqt4/assignment_loss‚5.468105763805156 /loss/FRt1/detection_loss‚5.8964084879213905 1loss/SRqqt4/detection_loss‚3.2677177783786036 &loss/total_loss‚123.34346118774978 0loss/FRt3/assignment_loss‚7.1553898192288585 1loss/SRqqt2/assignment_loss‚5.480413828955756 /loss/FBt1/assignment_loss‚2.185178844134013 1loss/SRqqt2/detection_loss‚3.2820643411742316 /loss/FBt4/detection_loss‚15.546235146847637 _step‚15 0loss/FRt1/assignment_loss‚7.1647231060525645 .loss/FRt3/detection_loss‚2.597114294505956 1loss/SRqqt3/detection_loss‚2.1750753177063804 _runtime‚ 153.075801657 0loss/FBt2/assignment_loss‚2.0839967250823976 /loss/FBt3/assignment_loss‚1.939698404556996 .loss/FRt4/detection_loss‚3.441218565492069 0loss/FBt4/assignment_loss‚2.1805224310268057 /loss/FBt3/detection_loss‚1.3004292618421165 trainer/global_step‚199 /loss/FRt2/assignment_loss‚7.387010050982964 .loss/FRt2/detection_loss‚4.031777743159271 0loss/SRqqt1/detection_loss‚4.829890804986159 % _timestamp‚1.7883955284390655e+09š ETóÎ .loss/FRt2/detection_loss‚4.031777743159271 0loss/SRqqt1/detection_loss‚4.829890804986159 % _timestamp‚1.7883955284390655e+09 1loss/SRqqt1/assignment_loss‚5.440170274840461 1loss/SRqqt3/assignment_loss‚5.302434369495937 /loss/FBt1/detection_loss‚11.202747503916422 .loss/FBt2/detection_loss‚6.563832863569259 epoch‚0 /loss/FRt4/assignment_loss‚7.421305460088393 1loss/SRqqt4/assignment_loss‚5.468105763805156 /loss/FRt1/detection_loss‚5.8964084879213905 1loss/SRqqt4/detection_loss‚3.2677177783786036 &loss/total_loss‚123.34346118774978 0loss/FRt3/assignment_loss‚7.1553898192288585 1loss/SRqqt2/assignment_loss‚5.480413828955756 /loss/FBt1/assignment_loss‚2.185178844134013 1loss/SRqqt2/detection_loss‚3.2820643411742316 /loss/FBt4/detection_loss‚15.546235146847637 _step‚15 0loss/FRt1/assignment_loss‚7.1647231060525645 .loss/FRt3/detection_loss‚2.597114294505956 1loss/SRqqt3/detection_loss‚2.1750753177063804 _runtime‚ 153.075801657 0loss/FBt2/assignment_loss‚2.0839967250823976 /loss/FBt3/assignment_loss‚1.939698404556996 .loss/FRt4/detection_loss‚3.441218565492069 0loss/FBt4/assignment_loss‚2.1805224310268057 /loss/FBt3/detection_loss‚1.3004292618421165 trainer/global_step‚199 /loss/FRt2/assignment_loss‚7.387010050982964‹ÇPãœô‚(:‘ ”€ãÔü—þ­ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚128.89' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.777999999999995 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èЊ-õ‚(:¢ ”€ãÔà«Ê° disk.drbd2266.out‚0 disk.drbd2265.out‚0 cpu‚25.17369964335778 disk.dm-1.out‚ 213.23828125 disk.dm-0.in‚0 disk.nvme0c0n1.in‚0$ disk.nvme0c0n1.out‚ 4.59130859375 disk.nvme0n1.in‚0$ memory_percent‚86.05036735534668 disk.rbd1.out‚ 0.619140625 disk.drbd2080.in‚0 disk.rbd0.out‚0" disk.nvme0n1.out‚ 4.59130859375 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda.in‚0 proc.cpu.threads‚17 network.recv‚147313 disk.drbd1716.in‚0 disk.sda1.in‚0 disk.sda.out‚ 213.23828125 disk.sda3.in‚0# proc.memory.rssMB‚ 2588.30078125 disk.rbd1.in‚0 network.sent‚266848 disk.sda2.out‚0 disk.dm-1.in‚0 disk.drbd2080.out‚0 disk.dm-0.out‚ 4.59130859375 disk.sda3.out‚ 213.23828125) disk./.usagePercent‚16.32020842781718% disk./.usageGB‚243.18289947509766 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0( proc.memory.availableMB‚ 2285.5078125* proc.memory.percent‚15.797734260559082×°Õö‚(:’ £€ãÔ¾Ù”® gpu.0.gpu‚28.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚136.974' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.394799999999996 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Gå&J.÷‚(:£ £€ãÔóîä° disk.sda2.in‚0 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 network.sent‚317217% disk./.usageGB‚243.18288040161133 disk.nvme0c0n1.in‚0$ disk.nvme0c0n1.out‚ 5.45751953125 disk.nvme0n1.in‚0" disk.nvme0n1.out‚ 5.45751953125 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 0.6513671875* proc.memory.percent‚15.797758102416992 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.dm-1.out‚ 230.18359375 disk.rbd0.out‚0 disk.dm-0.in‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 2312.6640625 cpu‚25.164591502541448) disk./.usagePercent‚16.32020714777951 disk.drbd2080.in‚0 network.recv‚153872 disk.sda.in‚0 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 230.18359375$ memory_percent‚85.88461875915527 disk.sda.out‚ 230.18359375 disk.dm-0.out‚ 5.45751953125 disk.rbd1.in‚0" proc.memory.rssMB‚ 2588.3046875âw¬ø+ _wandbruntime‚Þðœ189  _runtime‚189¼ÂÁgÆùÀ _runtime‚ 189.509933379 _step‚16 ' lr-AdamW/pg1‚4.4277640030087546e-05 ' lr-AdamW/pg2‚4.4277640030087546e-05 trainer/global_step‚249 % _timestamp‚1.7883955648730993e+09?_dÊúÄ ' lr-AdamW/pg1‚4.4277640030087546e-05 ' lr-AdamW/pg2‚4.4277640030087546e-05 trainer/global_step‚249 % _timestamp‚1.7883955648730993e+09 _runtime‚ 189.509933379 _step‚16þCMï0û+ _wandbruntime‚189  _runtime‚189â£x™'ü“O O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 353776483840 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4976051 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1852456 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 211351552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7177608608768 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 7593202639360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 346624595118 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 347622220800 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 6830004390912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚513 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 6495655508992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3173515 trainer/global_step‚249 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1852454 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 6830004390912 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 6829986387968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚972 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 _step‚17 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 347790453760 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 353987835392 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 346789708974 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 7239426155520 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4957511 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 7593437931008 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7177608608768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _runtime‚ 189.510828278 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3173515 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚993 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235291648 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 7177794844672 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3231012 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 7239450095616 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 347622220800 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚4957511 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 6842280104110 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7177794844672 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3231525 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚511 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 6842280104110 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 7177794844672 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3231525 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚511 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 HDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 353776483840 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚4976051 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚1852456 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 211351552 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 7177608608768 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 7593202639360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 346624595118 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 347622220800 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 6830004390912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚513 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 6495655508992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3173515 trainer/global_step‚249 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚1852454 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 6830004390912 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 6829986387968 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚972 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 _step‚17 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 347790453760 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 353987835392 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 346789708974 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚ 7239426155520 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚4957511 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 7593437931008 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 7177608608768 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _runtime‚ 189.510828278 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3173515 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚993 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235291648 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 7177794844672 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3231012 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚00LB#0þ+ _wandbruntime‚190  _runtime‚190|šðH£&ÿM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4977298 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 7268339103744 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 6857255752704 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 349183605248 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 349015358976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 349015358976 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7623537745408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 7206271111680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3243912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7206457360896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1859833 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3186176 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3186176 _runtime‚ 190.213004247 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 6857255752704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8182040 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8182040 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1384604 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8163472 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4995864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233181184 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 6521615730688 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3244437 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 7623770926592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4995864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 6521598691328 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 348013759662 % _timestamp‚1.7883955655757606e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 6857273755648 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 349183605248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 209241088 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _step‚18 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 355198641664 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 348178873726 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1859831 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4977298 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 7268363043840 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 trainer/global_step‚249ÈËû〡M SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 6521598691328 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 348013759662 % _timestamp‚1.7883955655757606e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚ 6857273755648 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 349183605248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 209241088 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _step‚18 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 355198641664 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 348178873726 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚1859831 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚4977298 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚ 7268363043840 E=DeviceStatsMonitor.on_train_baÔÒÇÛÄtch_end/segment.large_pool.peak‚217 trainer/global_step‚249 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚4977298 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 7268339103744 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 6857255752704 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 349183605248 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 349015358976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 349015358976 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 7623537745408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 Q?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚ 7206271111680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3243912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 7206457360896 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚1859833 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚3186176 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3186176 _runtime‚ 190.213004247 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 6857255752704 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚8182040 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚8182040 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1384604 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚8163472 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚4995864 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233181184 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚ 6521615730688 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚3244437 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 7623770926592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚4995864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1éGƒ0+ _wandbruntime‚190  _runtime‚190†âÑZF‚À /loss/FRt3/assignment_loss‚7.189048722105206 .loss/FRt3/detection_loss‚3.005314917778069 /loss/FBt1/detection_loss‚15.392905070023103 -loss/FBt2/detection_loss‚41.6998658105731 epoch‚0 1loss/SRqqt3/assignment_loss‚4.783103596674253 0loss/FBt1/assignment_loss‚1.3129686821590771 .loss/FRt2/detection_loss‚4.062633613260781 $ _timestamp‚1.788395565577288e+09 .loss/FRt1/assignment_loss‚7.18280901628382 1loss/SRqqt1/assignment_loss‚5.149706522623698 1loss/SRqqt2/assignment_loss‚4.917311337034581 0loss/FBt3/assignment_loss‚0.7341576028201315 .loss/FBt4/detection_loss‚8.000837194068092 _runtime‚ 190.214026426 _step‚19 /loss/FRt4/detection_loss‚3.4405613185143937 &loss/total_loss‚156.86916837385894 /loss/FRt2/assignment_loss‚7.430524849310154 /loss/FRt4/assignment_loss‚7.344721569734461 /loss/FBt4/assignment_loss‚1.096640842301505 .loss/FRt1/detection_loss‚8.005221079377566 0loss/SRqqt1/detection_loss‚9.666199096375042 0loss/SRqqt4/detection_loss‚4.479341620436082 .loss/SRqqt4/assignment_loss‚5.184791577168 0loss/FBt2/assignment_loss‚1.1856447458267212 .loss/FBt3/detection_loss‚1.258155244059809 1loss/SRqqt2/detection_loss‚2.4616383734648513 1loss/SRqqt3/detection_loss‚1.8850659718864584 trainer/global_step‚249>áfªJƒÄ trainer/global_step‚249 /loss/FRt3/assignment_loss‚7.189048722105206 .loss/FRt3/detection_loss‚3.005314917778069 /loss/FBt1/detection_loss‚15.392905070023103 -loss/FBt2/detection_loss‚41.6998658105731 epoch‚0 1loss/SRqqt3/assignment_loss‚4.783103596674253 0loss/FBt1/assignment_loss‚1.3129686821590771 .loss/FRt2/detection_loss‚4.062633613260781 $ _timestamp‚1.788395565577288e+09 .loss/FRt1/assignment_loss‚7.18280901628382 1loss/SRqqt1/assignment_loss‚5.149706522623698 1loss/SRqqt2/assignment_loss‚4.917311337034581 0loss/FBt3/assignment_loss‚0.7341576028201315 .loss/FBt4/detection_loss‚8.000837194068092 _runtime‚ 190.214026426 _step‚19 /loss/FRt4/detection_loss‚3.4405613185143937 &loss/total_loss‚156.86916837385894 /loss/FRt2/assignment_loss‚7.430524849310154 /loss/FRt4/assignment_loss‚7.344721569734461 /loss/FBt4/assignment_loss‚1.096640842301505 .loss/FRt1/detection_loss‚8.005221079377566 0loss/SRqqt1/detection_loss‚9.666199096375042 0loss/SRqqt4/detection_loss‚4.479341620436082 .loss/SRqqt4/assignment_loss‚5.184791577168 0loss/FBt2/assignment_loss‚1.1856447458267212 .loss/FBt3/detection_loss‚1.258155244059809 1loss/SRqqt2/detection_loss‚2.4616383734648513 1loss/SRqqt3/detection_loss‚1.8850659718864584 k–Ä’„‚(:‡ ²€ãԙؖ® gpu.0.gpu‚20.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚131.841' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.3682 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£ø]X.…‚(:£ ²€ãÔ ¢° disk.drbd1716.in‚0 disk.sda2.out‚0 disk.nvme0c0n1.in‚0* proc.memory.percent‚15.798044204711914 proc.cpu.threads‚17 disk.dm-1.in‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.dm-0.in‚0" disk.nvme0n1.out‚ 5.73486328125 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 246.81640625 network.sent‚323025 disk.sda.out‚ 246.81640625 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.dm-0.out‚ 5.73486328125 disk.sda3.out‚ 246.81640625 disk.nvme0n1.in‚0" proc.memory.rssMB‚ 2588.3515625 network.recv‚158101 disk.sda3.in‚0) proc.memory.availableMB‚ 2253.30859375 cpu‚25.15690066182517 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 0.6513671875$ disk.nvme0c0n1.out‚ 5.73486328125* disk./.usagePercent‚16.320208683824717% disk./.usageGB‚243.18290328979492 disk.sda.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0# memory_percent‚86.2468957901001ц‚(:’ Á€ãÔñ«Œ® gpu.0.gpu‚15.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚135.089' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.017799999999998 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡X)‡‚(:ž Á€ãÔõ„ì± disk.drbd2265.out‚0 disk.sda1.in‚0 disk.sda.out‚ 263.9453125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚0* proc.memory.percent‚15.798044204711914 network.sent‚372918 disk.dm-1.in‚0 disk.drbd2266.out‚0 disk.sda3.out‚ 263.9453125 disk.rbd1.in‚0 cpu‚25.14972536553337 disk.dm-1.out‚ 263.9453125" disk.nvme0n1.out‚ 6.15478515625 disk.sda.in‚0 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0$ disk.nvme0c0n1.out‚ 6.15478515625$ memory_percent‚86.24773025512695) disk./.usagePercent‚16.32020842781718 disk.sda1.out‚0" proc.memory.rssMB‚ 2588.3515625 proc.cpu.threads‚17 network.recv‚164141% disk./.usageGB‚243.18289947509766 disk.rbd0.out‚0 disk.rbd1.out‚ 0.8935546875 disk.nvme0c0n1.in‚0 disk.nvme0n1.in‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.dm-0.out‚ 6.15478515625' proc.memory.availableMB‚ 2253.171875 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.drbd1716.in‚0ã–Þ¦ˆ‚(:’ ЀãÔƒþø­ gpu.0.gpu‚34.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚119.341' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.868199999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÎOÛ)‰‚(:ž ЀãÔñ²Œ° disk.rbd1.in‚0" disk.nvme0n1.out‚ 176.517578125 disk.sda.out‚ 282.859375 network.recv‚168969 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.nvme0c0n1.in‚0 disk.sda2.out‚0 cpu‚25.14310584862163) disk./.usagePercent‚16.37029886196664 disk.sda.in‚0 disk.sda2.in‚0 disk.dm-0.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 282.859375 disk.nvme0n1.in‚0# proc.memory.rssMB‚ 2588.35546875 disk.drbd2080.in‚0 disk.dm-0.out‚ 176.517578125$ memory_percent‚86.24627590179443 proc.cpu.threads‚17% disk./.usageGB‚243.92928314208984 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 0.8935546875 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda3.in‚0$ disk.nvme0c0n1.out‚ 176.517578125) proc.memory.availableMB‚ 2253.41015625* proc.memory.percent‚15.798068046569824 network.sent‚379568 disk.dm-1.out‚ 282.859375 disk.drbd2266.in‚0 disk.drbd2265.in‚02s‡0Š+ _wandbruntime‚226  _runtime‚226qïÇÄ‹¾ trainer/global_step‚299 % _timestamp‚1.7883956022567787e+09 _runtime‚ 226.893625176 _step‚20 & lr-AdamW/pg1‚5.316873240560713e-05 & lr-AdamW/pg2‚5.316873240560713e-05eç‹ÈŒÂ _runtime‚ 226.893625176 _step‚20 & lr-AdamW/pg1‚5.316873240560713e-05 & lr-AdamW/pg2‚5.316873240560713e-05 trainer/global_step‚299 % _timestamp‚1.7883956022567787e+09HE™Ü0+ _wandbruntime‚226  _runtime‚226ÞHQÔ™'Ž“O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 424625625088 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3806465 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9775220 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 424828588032 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 416261527382 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5950213 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 417461776384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8610748168192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9109699191808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚993 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9109926094848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5968753 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 8193472627712 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3875372 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1654586 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 416096413526 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226903040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 trainer/global_step‚299 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8610748168192 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 417293543424 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3875892 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8685097506816 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 8193472627712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2221306 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3806467 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5950213 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚972 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 8208911039318 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7792831665152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7883956022576103e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚520 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9756678 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 8193454624768 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1654068 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 226.894873592 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3806465 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542£ÈýÁ;—O G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 WEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚ 8610934404096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 _step‚21 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 TBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚ 8610934404096 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚5968753 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 8209093192534 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 417461776384 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2221304 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚518 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 7792814625792 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 417293543424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 P>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚ 8610748168192 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 XFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚ 9109699191808 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚993 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 \JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚ 9109926094848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚5968753 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 8193472627712 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚3875372 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1654586 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 416096413526 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226903040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 trainer/global_step‚299 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 SADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚ 8610748168192 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 417293543424 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚3875892 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚ 8685097506816 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 8193472627712 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 HDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2221306 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚3806467 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚5950213 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚972 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 8208911039318 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 7792831665152 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 HDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7883956022576103e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚520 D8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚9756678 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 8193454624768 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1654068 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 226.894873592 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚3806465 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 424625625088 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚3806465 L@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚9775220 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 424828588032 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 416261527382 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚5950213 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 417461776384 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 8236243386198 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9140032433664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202949632 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 418854927872 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9789124 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2228683 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1659997 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 418686681600 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3819124 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚972 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚509 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 418686681600 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9789124 % _timestamp‚1.7883956029635267e+09 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3888171 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5970000 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9807692 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 8639596920320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5988566 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚993 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9140259323392 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7818757808128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2228681 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9807692 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5988566 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 426248868352 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _step‚22 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3819126 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3819124 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚ 227.601394144 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 418854927872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8713986514944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚507 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8639596920320®í„#§&’¡M L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚ 227.601394144 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 418854927872 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚ 8713986514944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚507 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 R@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚ 8639596920320 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 Q?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚ 8236243386198 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 VDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚ 9140032433664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202949632 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 418854927872 F:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚9789124 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2228683 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚1659997 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 418686681600 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚3819124 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚972 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚509 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 418686681600 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 B6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚9789124 % _timestamp‚1.7883956029635267e+09 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚3888171 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚5970000 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚9807692 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚ 8639596920320 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚5988566 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚993 ZHDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚ 9140259323392 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚ 7818757808128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2228681 F:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚9807692 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚5988566 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 426248868352 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _step‚22 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚3819126 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚3819124&6 0“+ _wandbruntime‚227  _runtime‚227 ÀL©N”È /loss/FRt2/assignment_loss‚7.311760890774611 epoch‚0 _runtime‚ 227.602636318 % _timestamp‚1.7883956029658558e+09 0loss/FRt1/assignment_loss‚7.0844810636419995 /loss/FRt3/assignment_loss‚7.152237717310587 1loss/FBt3/assignment_loss‚0.45262089750218776 0loss/SRqqt1/detection_loss‚4.511650676005765 0loss/SRqqt4/detection_loss‚4.286791962620459 &loss/total_loss‚117.69744867305072 _step‚23 .loss/FRt1/detection_loss‚5.938211936700012 1loss/SRqqt1/assignment_loss‚3.976021559614884 1loss/SRqqt2/assignment_loss‚4.020966688791911 /loss/FRt4/detection_loss‚3.0483128104293558 /loss/SRqqt2/detection_loss‚4.48339744284749 /loss/FBt2/detection_loss‚20.315952111035585 .loss/FBt3/detection_loss‚1.317177601357769 1loss/SRqqt4/assignment_loss‚4.009737799042149 0loss/FBt2/assignment_loss‚0.7146370382979512 1loss/SRqqt3/detection_loss‚1.1745118296086348 .loss/FRt2/detection_loss‚4.132459524201184 /loss/FBt1/detection_loss‚12.643911948570839 /loss/FRt4/assignment_loss‚7.338538579773485 1loss/SRqqt3/assignment_loss‚3.440575775347258 0loss/FBt1/assignment_loss‚0.5448420540644572 0loss/FBt4/assignment_loss‚0.4217249439097941 .loss/FBt4/detection_loss‚6.902995864550273 trainer/global_step‚299 .loss/FRt3/detection_loss‚2.473929957052072ŸV—R•Ì 0loss/SRqqt4/detection_loss‚4.286791962620459 &loss/total_loss‚117.69744867305072 _step‚23 .loss/FRt1/detection_loss‚5.938211936700012 1loss/SRqqt1/assignment_loss‚3.976021559614884 1loss/SRqqt2/assignment_loss‚4.020966688791911 /loss/FRt4/detection_loss‚3.0483128104293558 /loss/SRqqt2/detection_loss‚4.48339744284749 /loss/FBt2/detection_loss‚20.315952111035585 .loss/FBt3/detection_loss‚1.317177601357769 1loss/SRqqt4/assignment_loss‚4.009737799042149 0loss/FBt2/assignment_loss‚0.7146370382979512 1loss/SRqqt3/detection_loss‚1.1745118296086348 .loss/FRt2/detection_loss‚4.132459524201184 /loss/FBt1/detection_loss‚12.643911948570839 /loss/FRt4/assignment_loss‚7.338538579773485 1loss/SRqqt3/assignment_loss‚3.440575775347258 0loss/FBt1/assignment_loss‚0.5448420540644572 0loss/FBt4/assignment_loss‚0.4217249439097941 .loss/FBt4/detection_loss‚6.902995864550273 trainer/global_step‚299 .loss/FRt3/detection_loss‚2.473929957052072 /loss/FRt2/assignment_loss‚7.311760890774611 epoch‚0 _runtime‚ 227.602636318 % _timestamp‚1.7883956029658558e+09 0loss/FRt1/assignment_loss‚7.0844810636419995 /loss/FRt3/assignment_loss‚7.152237717310587 1loss/FBt3/assignment_loss‚0.45262089750218776 0loss/SRqqt1/detection_loss‚4.511650676005765vÕÚ’–‚(:‡ ߀ãÔȉ¶® gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚125.932' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.1864 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ª¾…÷S—‚(:È ߀ãÔç⑱ disk.rbd1.out‚ 0.92578125 disk.nvme0c0n1.in‚ 14.9609375 disk.drbd2266.out‚0 cpu‚25.138571863456388 network.recv‚174125 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0) proc.memory.availableMB‚ 2253.29296875* proc.memory.percent‚15.798068046569824% disk./.usageGB‚249.25280380249023 disk.sda.in‚0' disk.nvme0c0n1.out‚4011.68603515625% disk.nvme0n1.out‚4011.68603515625 disk.sda2.out‚0 disk.dm-1.in‚0 disk.sda1.in‚0$ memory_percent‚86.24699115753174 proc.cpu.threads‚17 network.sent‚386085* disk./.usagePercent‚16.727564800217454 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 297.66015625 disk.drbd2265.out‚0 disk.rbd0.out‚0" disk.dm-0.out‚4008.81103515625# proc.memory.rssMB‚ 2588.35546875 disk.sda.out‚ 297.66015625 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 14.9609375 disk.rbd1.in‚0 disk.nvme0n1.in‚ 14.9609375 disk.dm-1.out‚ 297.66015625 disk.sda1.out‚0š:‹’˜‚(:‡ î€ãÔýä•® gpu.0.gpu‚30.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚127.528' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.5056 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}t,ÊK™‚(:À î€ãÔžä° disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 14.9609375 disk.sda3.out‚ 313.90234375* proc.memory.percent‚15.798091888427734 proc.cpu.threads‚17 network.sent‚435633$ disk./.usageGB‚251.7905044555664 disk.sda.out‚ 313.90234375 disk.rbd1.out‚ 1.01171875 disk.sda.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 313.90234375 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚7172.19384765625 cpu‚25.13389598419766 disk.sda2.in‚0 disk.dm-1.in‚0 disk.nvme0c0n1.in‚ 14.9609375! proc.memory.rssMB‚ 2588.359375 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0& proc.memory.availableMB‚ 2253.28125 network.recv‚179378) disk./.usagePercent‚16.89787202031807" disk.dm-0.out‚7172.19384765625' disk.nvme0c0n1.out‚7172.19384765625 disk.nvme0n1.in‚ 14.9609375$ memory_percent‚86.24706268310547 disk.drbd1716.out‚0§“™ˆ0š+ _wandbruntime‚264  _runtime‚264ìÄ›¾ trainer/global_step‚349 % _timestamp‚1.7883956395613048e+09 _runtime‚ 264.198181197 _step‚24 & lr-AdamW/pg1‚6.205982478112672e-05 & lr-AdamW/pg2‚6.205982478112672e-05F.JõȜ % _timestamp‚1.7883956395613048e+09 _runtime‚ 264.198181197 _step‚24 & lr-AdamW/pg1‚6.205982478112672e-05 & lr-AdamW/pg2‚6.205982478112672e-05 trainer/global_step‚349Ý¥‡M0+ _wandbruntime‚264  _runtime‚264J¸‘°6 žŸO ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 9556922861568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1928633 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4518245 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6961455 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4439415 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 9575541974526 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 485568231934 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚972 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚10626272435712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2590156 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 9089973742592 % _timestamp‚1.7883956395620518e+09 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 211351552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6961455 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4518789 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚ÍË{ëo0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 9556940864512 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11382330 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 9556940864512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4439417 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚10043887727616 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2590154 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 495762809344 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 485733345790 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚10626507727360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚10130744918016 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚10130720977920 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 487133099008 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6942915 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚993 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 _step‚25 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚10043887727616 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 486964866048 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11400872 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 495551457792 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _runtime‚ 264.199132182 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4439417 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 487133099008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚10044073963520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235291648 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 9575724127742 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4439415 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 9556922861568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚10044073963520 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 486964866048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 9089990781952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 trainer/global_step‚349 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1928091 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416¸ÑØ«©'Ÿ£O RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚4518245 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚6961455 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚4439415 SADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚ 9575541974526 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 485568231934 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚972 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚10626272435712 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2590156 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚ 9089973742592 % _timestamp‚1.7883956395620518e+09 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 211351552 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚6961455 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚4518789 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚ 9556940864512 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚11382330 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚ 9556940864512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚4439417 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚10043887727616 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚2590154 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 495762809344 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 485733345790 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚10626507727360 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚10130744918016 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚10130720977920 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 487133099008 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚6942915 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚993 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 _step‚25 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚10043887727616 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 486964866048 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚11400872 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 495551457792 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _runtime‚ 264.199132182 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚4439417 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 487133099008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚10044073963520 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235291648 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚ 9575724127742 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚4439415 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚ 9556922861568 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚10044073963520 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 486964866048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚ 9089990781952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 trainer/global_step‚349 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚1928091 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚ 9556922861568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚1928633 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540ø)/Ä0 + _wandbruntime‚264  _runtime‚264­­òµ¯&¡©M D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6962702 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11433344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4531480 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚993 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚10656839972864 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 488358004224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 496966574080 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4530972 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚506 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10656600500224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4452074 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚26 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11414776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10159657866240 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 488358004224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2597533 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 487122510542 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11414776 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 9584210229248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _runtime‚ 264.907042239 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 trainer/global_step‚349 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4452076 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 9584192226304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11433344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 486957396478 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 488526250496 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4452074 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 239472640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚508 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2597531 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 9584192226304 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 497182106624 % _timestamp‚1.7883956402694743e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1933441 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 215532544*,‰æ¢­M MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚4452076 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚ 9584192226304 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚11433344 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 486957396478 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 488526250496 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚4452074 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 239472640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚508 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2597531 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚ 9584192226304 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 497182106624 % _timestamp‚1.7883956402694743e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚1933441 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 215532544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚6962702 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚11433344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚4531480 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚993 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚10656839972864 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 488358004224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 496966574080 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚4530972 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚506 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚10656600500224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚4452074 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚26 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚11414776 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚10159657866240 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 488358004224 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2597533 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 487122510542 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚11414776 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚ 9584210229248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _runtime‚ 264.907042239 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 trainer/global_step‚349W4¤0£+ _wandbruntime‚264  _runtime‚264i~:O¤É /loss/FRt2/detection_loss‚4.2322888292372225 _step‚27 /loss/FRt2/assignment_loss‚7.370099103450775 2loss/SRqqt2/assignment_loss‚3.7564965713591802 0loss/FBt3/assignment_loss‚0.3088025720207952 .loss/FRt1/detection_loss‚4.569842967493781 1loss/SRqqt4/detection_loss‚2.8184347361195505 .loss/FRt4/assignment_loss‚7.33170104513363 0loss/FBt4/assignment_loss‚0.6891077821264449 .loss/FBt2/detection_loss‚16.11784557104111 _runtime‚ 264.908170638 0loss/FBt1/assignment_loss‚0.4761703314259648 /loss/FRt3/detection_loss‚2.4091123347244565 0loss/SRqqt1/detection_loss‚3.522481002534429 1loss/SRqqt3/detection_loss‚0.8891684240645614 &loss/total_loss‚106.39113273641149 /loss/FRt3/assignment_loss‚7.233594410003177 0loss/SRqqt3/assignment_loss‚3.41943340432154 .loss/FBt4/detection_loss‚7.071603554746379 2loss/SRqqt4/assignment_loss‚3.6550363281316924 1loss/SRqqt2/detection_loss‚3.6109481117732467 /loss/FBt1/detection_loss‚10.994074100255967 /loss/FBt3/detection_loss‚1.0830136565961368 % _timestamp‚1.7883956402713828e+09 /loss/FRt1/assignment_loss‚7.301905270280509 0loss/FBt2/assignment_loss‚0.5282415691763163 .loss/FRt4/detection_loss‚3.485177847195645 0loss/SRqqt1/assignment_loss‚3.51655321319898 epoch‚0 trainer/global_step‚349'Û—±S¥Í % _timestamp‚1.7883956402713828e+09 /loss/FRt1/assignment_loss‚7.301905270280509 0loss/FBt2/assignment_loss‚0.5282415691763163 .loss/FRt4/detection_loss‚3.485177847195645 0loss/SRqqt1/assignment_loss‚3.51655321319898 epoch‚0 trainer/global_step‚349 /loss/FRt2/detection_loss‚4.2322888292372225 _step‚27 /loss/FRt2/assignment_loss‚7.370099103450775 2loss/SRqqt2/assignment_loss‚3.7564965713591802 0loss/FBt3/assignment_loss‚0.3088025720207952 .loss/FRt1/detection_loss‚4.569842967493781 1loss/SRqqt4/detection_loss‚2.8184347361195505 .loss/FRt4/assignment_loss‚7.33170104513363 0loss/FBt4/assignment_loss‚0.6891077821264449 .loss/FBt2/detection_loss‚16.11784557104111 _runtime‚ 264.908170638 0loss/FBt1/assignment_loss‚0.4761703314259648 /loss/FRt3/detection_loss‚2.4091123347244565 0loss/SRqqt1/detection_loss‚3.522481002534429 1loss/SRqqt3/detection_loss‚0.8891684240645614 &loss/total_loss‚106.39113273641149 /loss/FRt3/assignment_loss‚7.233594410003177 0loss/SRqqt3/assignment_loss‚3.41943340432154 .loss/FBt4/detection_loss‚7.071603554746379 2loss/SRqqt4/assignment_loss‚3.6550363281316924 1loss/SRqqt2/detection_loss‚3.6109481117732467 /loss/FBt1/detection_loss‚10.994074100255967 /loss/FBt3/detection_loss‚1.0830136565961368 ¥ª’¦‚(:‡ ý€ãÔþ¶É® gpu.0.gpu‚33.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚118.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7332 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇÛ&6§‚(:« ý€ãԠű disk.drbd2080.out‚0 disk.dm-0.in‚ 147.0859375 disk.drbd1716.out‚0 disk.sda3.in‚0( proc.memory.availableMB‚ 2252.7265625 cpu‚25.129545886584065$ disk.nvme0n1.out‚9006.1005859375 network.sent‚441845% disk./.usageGB‚251.87259674072266 disk.sda3.out‚332 disk.dm-1.out‚332) disk./.usagePercent‚16.90338130245492 disk.rbd0.out‚0! disk.dm-0.out‚9006.1005859375 disk.rbd1.out‚ 1.16796875 disk.nvme0n1.in‚ 147.0859375$ memory_percent‚86.25044822692871 network.recv‚183802 disk.rbd1.in‚0 disk.sda.out‚332 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.sda.in‚0# proc.memory.rssMB‚ 2588.36328125* proc.memory.percent‚15.798115730285645 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0c0n1.out‚9006.1005859375 disk.dm-1.in‚0 disk.drbd2266.out‚0! disk.nvme0c0n1.in‚ 147.0859375 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.drbd2265.out‚0,M¶¨‚(:… ŒãÔµƒµ® gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚109.22' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.844 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µçÎ’R©‚(:Ç ŒãÔ©Àª± network.sent‚491822 disk.sda3.out‚ 357.98046875" disk.nvme0c0n1.in‚ 167.07421875 disk.rbd0.out‚0$ memory_percent‚86.25125885009766" proc.memory.rssMB‚ 2588.3671875 cpu‚25.125680144492176% disk./.usageGB‚251.88397979736328 disk.rbd1.out‚ 1.16796875& disk.nvme0c0n1.out‚9006.2685546875 disk.sda2.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd2265.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0! disk.dm-0.out‚9006.2685546875 disk.sda.in‚0 disk.dm-1.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0& proc.memory.availableMB‚ 2252.59375* disk./.usagePercent‚16.904145228937093 disk.sda3.in‚0 disk.nvme0n1.in‚ 167.07421875$ disk.nvme0n1.out‚9006.2685546875 disk.sda.out‚ 357.98046875 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 network.recv‚189684 disk.drbd2265.out‚0 disk.dm-0.in‚ 167.07421875 disk.sda1.out‚0 disk.dm-1.out‚ 357.98046875 disk.sda1.in‚0* proc.memory.percent‚15.798139572143555FÛ3D’ª‚(:‡ ›ãԻ鷯 gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚115.419' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0838 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕƒëkS«‚(:È ›ãÔ­Ÿä± network.sent‚497467 disk.drbd2080.in‚0% disk.nvme0n1.out‚9098.96630859375 disk.drbd2265.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 396.859375* proc.memory.percent‚15.798139572143555' disk.nvme0c0n1.out‚9098.96630859375 disk.drbd2265.in‚0 network.recv‚194190% disk./.usageGB‚251.85002899169922 disk.sda.in‚0 disk.dm-1.out‚ 396.859375$ memory_percent‚86.24567985534668( proc.memory.availableMB‚ 2253.5078125 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 167.07421875" proc.memory.rssMB‚ 2588.3671875 cpu‚25.122113488231818 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚ 1.2001953125 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 167.07421875 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.dm-1.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0" disk.dm-0.out‚9098.96630859375* disk./.usagePercent‚16.901866761882353" disk.nvme0c0n1.in‚ 167.07421875 disk.sda.out‚ 396.859375HÿE0¬+ _wandbruntime‚301  _runtime‚301ß6}¦Â­¼ % lr-AdamW/pg1‚7.09509171566463e-05 % lr-AdamW/pg2‚7.09509171566463e-05 trainer/global_step‚399 % _timestamp‚1.7883956770471754e+09 _runtime‚ 301.684072203 _step‚28Á `Æ®À _runtime‚ 301.684072203 _step‚28 % lr-AdamW/pg1‚7.09509171566463e-05 % lr-AdamW/pg2‚7.09509171566463e-05 trainer/global_step‚399 % _timestamp‚1.7883956770471754e+09çâšþ0¯+ _wandbruntime‚301  _runtime‚301x!P¬­'°§O QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚507 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚2959006 OCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚7954157 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 555205164198 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7935617 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11477027287040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13007982 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7954157 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5072365 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚10920409101312 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5072367 _step‚29 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚11477213522944 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 566677320192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚993 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚ 301.685068283 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚509 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 566472260096 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 555205164198 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚7935617 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚11477027287040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚13007982 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚7954157 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5072365 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚10920409101312 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5072367 _step‚29 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚11477213522944 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 566677320192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚993 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _runtime‚ 301.685068283 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚509 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 566472260096 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor¦ÔÆïÕ$.on_train_batch_end/allocated_bytes.large_pool.allocated‚10947678466048 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2966383 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 568096118784 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 556429214886 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13040428 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚10413076041728 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13040428 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚10947660463104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2209926 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12173401396224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚11505876039168 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 558029326848 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7955404 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 558029326848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚10969505256614 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11605281337344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 # _timestamp‚1.78839567773936e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5176309 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚993 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 207143936 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 trainer/global_step‚399 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5085026 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7973970 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5175811 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5085026 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5085024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 _step‚30 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚10947678466048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 556594328950 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 558197573120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚498 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚11605305277440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2966381 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 _runtime‚ 302.3767798 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13058996 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2209430 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚13040428 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚10413076041728 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚13040428 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚10947660463104 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2209926 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚12173401396224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚11505876039168 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 558029326848 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚7955404 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 558029326848 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚10969505256614 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚11605281337344 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 # _timestamp‚1.78839567773936e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5176309 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚993 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 207143936 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 trainer/global_step‚399 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚5085026 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚7973970 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5175811 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚5085026 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5085024 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 _step‚30 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚10947678466048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 556594328950 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 558197573120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚498 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚11605305277440 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚2966381 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 _runtime‚ 302.3767798 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚13058996 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2209430 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚10947678466048 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚2966383 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 568096118784Ý8Ê0µ+ _wandbruntime‚302  _runtime‚3028W2S¶Í /loss/FRt3/assignment_loss‚7.207013957354487 /loss/FRt3/detection_loss‚2.9953950176737747 2loss/SRqqt4/assignment_loss‚3.5041940435767174 /loss/FRt1/assignment_loss‚7.287207946181297 0loss/FBt4/assignment_loss‚0.5237855906324351 1loss/SRqqt2/detection_loss‚2.4366783351383425 0loss/SRqqt3/detection_loss‚0.802237809672591 /loss/FBt3/detection_loss‚1.1963348204692983 $ _timestamp‚1.788395677741073e+09 /loss/FRt2/assignment_loss‚7.462333815438407 0loss/FBt1/assignment_loss‚0.2983261920910861 .loss/FRt1/detection_loss‚4.641458901576698 0loss/SRqqt1/detection_loss‚5.660433106621107 .loss/FBt1/detection_loss‚11.69723328948021 /loss/FBt2/detection_loss‚158.65578252077103 trainer/global_step‚399 2loss/SRqqt1/assignment_loss‚3.6822675784428913 2loss/SRqqt2/assignment_loss‚3.6082672769373114 2loss/FBt2/assignment_loss‚0.008134094998240471 1loss/FBt3/assignment_loss‚0.48087051291284816 .loss/FBt4/detection_loss‚9.750697321751538 /loss/FRt4/assignment_loss‚7.340859936732872 2loss/SRqqt3/assignment_loss‚3.3464632403682657 _runtime‚ 302.377848788 .loss/FRt2/detection_loss‚5.912028817193849 &loss/total_loss‚254.23553505888202 _step‚31 .loss/FRt4/detection_loss‚3.300950136839175 0loss/SRqqt4/detection_loss‚2.436580796027556 epoch‚0 •—ZW·Ñ 1loss/SRqqt2/detection_loss‚2.4366783351383425 0loss/SRqqt3/detection_loss‚0.802237809672591 /loss/FBt3/detection_loss‚1.1963348204692983 $ _timestamp‚1.788395677741073e+09 /loss/FRt2/assignment_loss‚7.462333815438407 0loss/FBt1/assignment_loss‚0.2983261920910861 .loss/FRt1/detection_loss‚4.641458901576698 0loss/SRqqt1/detection_loss‚5.660433106621107 .loss/FBt1/detection_loss‚11.69723328948021 /loss/FBt2/detection_loss‚158.65578252077103 trainer/global_step‚399 2loss/SRqqt1/assignment_loss‚3.6822675784428913 2loss/SRqqt2/assignment_loss‚3.6082672769373114 2loss/FBt2/assignment_loss‚0.008134094998240471 1loss/FBt3/assignment_loss‚0.48087051291284816 .loss/FBt4/detection_loss‚9.750697321751538 /loss/FRt4/assignment_loss‚7.340859936732872 2loss/SRqqt3/assignment_loss‚3.3464632403682657 _runtime‚ 302.377848788 .loss/FRt2/detection_loss‚5.912028817193849 &loss/total_loss‚254.23553505888202 _step‚31 .loss/FRt4/detection_loss‚3.300950136839175 0loss/SRqqt4/detection_loss‚2.436580796027556 epoch‚0 /loss/FRt3/assignment_loss‚7.207013957354487 /loss/FRt3/detection_loss‚2.9953950176737747 2loss/SRqqt4/assignment_loss‚3.5041940435767174 /loss/FRt1/assignment_loss‚7.287207946181297 0loss/FBt4/assignment_loss‚0.5237855906324351ß[¾‘¸‚(:† ªãԬת® gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚116.485' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.297 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å…R£G¹‚(:¼ ªãÔ›ÀÓ°" disk.dm-0.out‚9104.56787109375 network.recv‚198094 disk.sda2.in‚0 disk.sda3.in‚0$ memory_percent‚86.24451160430908) proc.memory.availableMB‚ 2253.69921875" proc.memory.rssMB‚ 2588.3671875 disk.rbd1.out‚ 1.2001953125* disk./.usagePercent‚16.901460989940585 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 167.07421875 network.sent‚503210$ disk./.usageGB‚251.8439826965332 disk.sda.in‚0 disk.dm-1.out‚425.75 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.sda3.out‚425.75 disk.sda.out‚425.75 disk.dm-1.in‚0" disk.nvme0c0n1.in‚ 167.07421875 disk.nvme0n1.in‚ 167.07421875* proc.memory.percent‚15.798139572143555 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚9104.56787109375 cpu‚25.118370515272733 disk.sda1.out‚0 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚9104.56787109375 disk.drbd2265.out‚0Û ‰hº‚(:’ ¹ãÔËñ® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚128.064' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.612799999999996 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î¢v“W»‚(:Ì ¹ãÔò׿±* disk./.usagePercent‚16.901460989940585" disk.nvme0c0n1.in‚ 167.07421875 disk.sda1.out‚0 disk.rbd0.out‚0" disk.dm-0.out‚9105.04345703125 disk.sda3.in‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 network.recv‚203531 disk.nvme0n1.in‚ 167.07421875 disk.sda.out‚ 484.3515625 disk.drbd2266.out‚0 disk.drbd2265.in‚0 network.sent‚553114 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.sda3.out‚ 484.3515625 disk.rbd1.in‚0 disk.rbd1.out‚ 1.4423828125) proc.memory.availableMB‚ 2253.66015625 proc.cpu.threads‚17 disk.sda2.out‚0 disk.sda.in‚0 disk.drbd2266.in‚0 disk.sda1.in‚0$ memory_percent‚86.24475002288818# proc.memory.rssMB‚ 2588.37109375 disk.dm-1.out‚ 484.3515625 disk.dm-0.in‚ 167.07421875* proc.memory.percent‚15.798163414001465' disk.nvme0c0n1.out‚9105.04345703125% disk.nvme0n1.out‚9105.04345703125 disk.dm-1.in‚0 cpu‚25.115356575875236$ disk./.usageGB‚251.8439826965332 disk.drbd2080.in‚0 disk.rbd0.in‚0ãÝÊç0¼+ _wandbruntime‚339  _runtime‚339MJa½¼ % lr-AdamW/pg1‚7.98420095321659e-05 % lr-AdamW/pg2‚7.98420095321659e-05 trainer/global_step‚449 % _timestamp‚1.7883957145031624e+09 _runtime‚ 339.140002108 _step‚32¶YÛƾÀ % lr-AdamW/pg1‚7.98420095321659e-05 % lr-AdamW/pg2‚7.98420095321659e-05 trainer/global_step‚449 % _timestamp‚1.7883957145031624e+09 _runtime‚ 339.140002108 _step‚32 LÀA\0¿+ _wandbruntime‚339  _runtime‚339â{™W„À§O OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5705317 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚12283877338112 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3327854 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13659552647680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 624511868750 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8928319 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚12283859335168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 % _timestamp‚1.7883957145038784e+09 _step‚33 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5806155 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3327856 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚12308985998158 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12910166846464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5805605 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14633634 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5705315 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚12283859335168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚12283877338112 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5705317 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235291648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11684309015552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8946859 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚12910166846464 UGDeviceStatsMonitor.on_train_batch_sàÓtì) tart/requested_bytes.small_pool.peak‚ 406782421 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5705315 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 637301555712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚13659317356032 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 626475744256 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 211351552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0K=±'Á«O ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 626475744256 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 211351552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚5705317 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚12283877338112 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3327854 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚13659552647680 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 624511868750 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚8928319 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚12283859335168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 % _timestamp‚1.7883957145038784e+09 _step‚33 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚5806155 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3327856 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚12308985998158 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚12910166846464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚5805605 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚14633634 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚5705315 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚12283859335168 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚12283877338112 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚5705317 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235291648 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚11684309015552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚8946859 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚12910166846464 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚5705315 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 637301555712 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚13659317356032 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199!=èQ0Â+ _wandbruntime‚339  _runtime‚339‰%!š·&ñM G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚14684648 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚12311146702848 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚972 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 638930977792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚8948106 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 627700649472 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚12311128699904 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 627868895744 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3335231 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5819109 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚993 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8948106 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 % _timestamp‚1.7883957152208297e+09 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚13689883666432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚34 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚12939015598592 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5818590 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 626066147358 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 339.858382739 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5717974 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14684648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚12336318345246 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚449 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11710252197888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2483359 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚12336136191822 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8966672 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚12938829349376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 237375488 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3335233 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚13050952688640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ODeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚12311128699904 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 627868895744 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3335231 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚5819109 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚993 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚8948106 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 % _timestamp‚1.7883957152208297e+09 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚13689883666432 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚34 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚12939015598592 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚5818590 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 626066147358 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 339.858382739 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚5717974 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚14684648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚12336318345246 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 trainer/global_step‚449 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚11710252197888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2483359 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚12336136191822 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚8966672 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚12938829349376 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 237375488 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3335233 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚13050952688640 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408"GÞa”0Å+ _wandbruntime‚339  _runtime‚339 Ã,œPÆÊ /loss/FRt3/assignment_loss‚7.162007493972778 1loss/SRqqt3/assignment_loss‚3.071535913149516 /loss/FBt4/assignment_loss‚0.547360458644107 /loss/FBt3/detection_loss‚0.8637364797814875 2loss/SRqqt4/assignment_loss‚3.1657477041770674 epoch‚0 trainer/global_step‚449 .loss/FRt4/assignment_loss‚7.40351098482726 0loss/SRqqt4/detection_loss‚2.514913890703485 /loss/FRt1/assignment_loss‚7.312635529425837 0loss/FBt2/assignment_loss‚0.5690399408922531 1loss/FBt3/assignment_loss‚0.48803522746617856 _step‚35 % _timestamp‚1.7883957152227442e+09 /loss/FRt2/detection_loss‚3.9322439276036762 /loss/FRt3/detection_loss‚2.8221037651225926 .loss/FBt1/detection_loss‚5.567581267229149 .loss/FBt4/detection_loss‚4.583398447426803 1loss/SRqqt2/assignment_loss‚3.480007540095936 0loss/SRqqt1/detection_loss‚5.421837830735791 1loss/SRqqt2/detection_loss‚2.4380441438068043 1loss/SRqqt3/detection_loss‚1.4817028739542972 .loss/FBt2/detection_loss‚19.42556831985712 2loss/SRqqt1/assignment_loss‚3.1938214763518302 0loss/FBt1/assignment_loss‚0.4506581744977406 /loss/FRt4/detection_loss‚2.6444262256387803 &loss/total_loss‚100.10505107775583 _runtime‚ 339.859530264 /loss/FRt2/assignment_loss‚7.259693123045421 .loss/FRt1/detection_loss‚4.305440339349931¥j9úTÇÎ /loss/FRt1/assignment_loss‚7.312635529425837 0loss/FBt2/assignment_loss‚0.5690399408922531 1loss/FBt3/assignment_loss‚0.48803522746617856 _step‚35 % _timestamp‚1.7883957152227442e+09 /loss/FRt2/detection_loss‚3.9322439276036762 /loss/FRt3/detection_loss‚2.8221037651225926 .loss/FBt1/detection_loss‚5.567581267229149 .loss/FBt4/detection_loss‚4.583398447426803 1loss/SRqqt2/assignment_loss‚3.480007540095936 0loss/SRqqt1/detection_loss‚5.421837830735791 1loss/SRqqt2/detection_loss‚2.4380441438068043 1loss/SRqqt3/detection_loss‚1.4817028739542972 .loss/FBt2/detection_loss‚19.42556831985712 2loss/SRqqt1/assignment_loss‚3.1938214763518302 0loss/FBt1/assignment_loss‚0.4506581744977406 /loss/FRt4/detection_loss‚2.6444262256387803 &loss/total_loss‚100.10505107775583 _runtime‚ 339.859530264 /loss/FRt2/assignment_loss‚7.259693123045421 .loss/FRt1/detection_loss‚4.305440339349931 /loss/FRt3/assignment_loss‚7.162007493972778 1loss/SRqqt3/assignment_loss‚3.071535913149516 /loss/FBt4/assignment_loss‚0.547360458644107 /loss/FBt3/detection_loss‚0.8637364797814875 2loss/SRqqt4/assignment_loss‚3.1657477041770674 epoch‚0 trainer/global_step‚449 .loss/FRt4/assignment_loss‚7.40351098482726 0loss/SRqqt4/detection_loss‚2.514913890703485#¶G…~È‚(:’ ÈãÔ¹„­® gpu.0.gpu‚32.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚115.956' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.191200000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a:uNÉ‚(:à ÈãÔÓÁ÷±" disk.nvme0c0n1.in‚ 167.07421875 disk.nvme0n1.in‚ 167.07421875 proc.memory.rssMB‚2588.375 disk.dm-1.in‚0 disk.drbd2265.in‚0 disk.sda.out‚ 517.078125" disk.dm-0.out‚9105.27392578125 disk.rbd1.in‚0 disk.rbd1.out‚ 1.4423828125 network.sent‚559557 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.sda.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 167.07421875) proc.memory.availableMB‚ 2253.61328125 disk.drbd2265.out‚0 network.recv‚208171* disk./.usagePercent‚16.901460989940585 disk.rbd0.out‚0 disk.sda3.out‚ 517.078125# memory_percent‚86.2450361251831 cpu‚25.112646078051007$ disk./.usageGB‚251.8439826965332 disk.sda2.in‚0 disk.dm-1.out‚ 517.078125 disk.sda3.in‚0* proc.memory.percent‚15.798187255859375 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0' disk.nvme0c0n1.out‚9105.27392578125% disk.nvme0n1.out‚9105.27392578125 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.sda1.in‚0m[(Ê‚(:… ×ãÔÚ¥¦® gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚105.98' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.196 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‚yØyQË‚(:Æ ×ãÔÝÀ„±* proc.memory.percent‚15.798187255859375 cpu‚25.110975394065708 network.recv‚214691 disk.drbd2080.out‚0" disk.dm-0.out‚9106.48779296875 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.sda3.in‚0 disk.sda3.out‚ 567.4921875 disk.rbd1.out‚ 1.474609375' disk.nvme0c0n1.out‚9106.48779296875 proc.memory.rssMB‚2588.375 disk.dm-1.out‚ 567.4921875 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 167.07421875$ disk./.usageGB‚251.8439826965332 disk.drbd2266.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 167.07421875) proc.memory.availableMB‚ 2253.31640625 network.sent‚609831 disk.sda.out‚ 567.4921875 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.dm-1.in‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚9106.48779296875$ memory_percent‚86.24684810638428 proc.cpu.threads‚17* disk./.usagePercent‚16.901460989940585 disk.rbd0.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 167.07421875 disk.sda.in‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0ÀDú’Ì‚(:‡ æãÔ¾½¡® gpu.0.gpu‚17.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚134.263' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.8526 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h™.PÍ‚(:Å æãÔø©¶± network.sent‚615978 disk.rbd0.in‚0% disk.nvme0n1.out‚9106.70263671875) disk./.usagePercent‚16.90146124594812% disk./.usageGB‚251.84398651123047 disk.drbd2080.out‚0 disk.sda2.in‚0* proc.memory.percent‚15.798187255859375 disk.drbd2266.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 605.6953125 disk.nvme0n1.in‚ 167.07421875$ memory_percent‚86.24541759490967 disk.dm-1.out‚ 605.6953125 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0' disk.nvme0c0n1.out‚9106.70263671875) proc.memory.availableMB‚ 2253.55078125 disk.dm-0.in‚ 167.07421875" disk.dm-0.out‚9106.70263671875 disk.rbd1.out‚ 1.560546875 disk.sda.in‚0 disk.sda.out‚ 605.6953125 disk.dm-1.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 cpu‚25.10801283537248 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 167.07421875 proc.memory.rssMB‚2588.375 network.recv‚218866 disk.sda2.out‚0 proc.cpu.threads‚17Ž„‹0Î+ _wandbruntime‚377  _runtime‚377a!š½ÄϾ & lr-AdamW/pg1‚8.873310190768548e-05 & lr-AdamW/pg2‚8.873310190768548e-05 trainer/global_step‚499 % _timestamp‚1.7883957524068022e+09 _runtime‚ 377.043636807 _step‚36–•À-ÈР& lr-AdamW/pg2‚8.873310190768548e-05 trainer/global_step‚499 % _timestamp‚1.7883957524068022e+09 _runtime‚ 377.043636807 _step‚36 & lr-AdamW/pg1‚8.873310190768548e-05$KI•’0Ñ+ _wandbruntime‚377  _runtime‚377lÇ ­'Ò§O XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16277828 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3696704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚14467687151616 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2752030 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 693983687158 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚13647327571968 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14343492641792 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚972 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6338265 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15176172463104 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 708271862784 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14343306405888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚13675434780150 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3696706 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚15175935074304 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚12981451092992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9921021 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6338265 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 % _timestamp‚1.7883957524076324e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚13675616933366 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚13647345574912 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9939561 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 694148801014 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16259286 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160tTQ±'Ó«O XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚13647327571968 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚14343492641792 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚972 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚6338265 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚15176172463104 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 708271862784 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚14343306405888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚13675434780150 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚3696706 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚15175935074304 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚12981451092992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 K?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚9921021 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6338265 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 % _timestamp‚1.7883957524076324e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚13675616933366 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚13647345574912 SGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚9939561 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 694148801014 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚16259286 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚16277828 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚3696704 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚14467687151616 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚2752030 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 693983687158 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944%ûiy…0Ô+ _wandbruntime‚377  _runtime‚377\âúçÕ°M VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚13007411314688 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9959374 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2757221 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6350924 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14372155158016 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 697540218368 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚14372155158016 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚13674596936704 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 trainer/global_step‚499 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16310300 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚993 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231084032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9959374 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14371968908800 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6461302 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_trPx$wÏain_batch_end/inactive_split.large_pool.allocated‚3704083 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9940808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚972 _step‚38 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6461832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.788395753109248e+09 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15206268187136 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 695537965766 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2757749 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3704081 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚13007394275328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13674596936704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9940808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6350926 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 697540218368 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 697371972096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14496576159744 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚13702949280454³Ú¶º&Ö´M ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚3704083 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 I=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚9940808 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚972 _step‚38 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚6461832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 $ _timestamp‚1.788395753109248e+09 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚15206268187136 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 695537965766 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚2757749 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚3704081 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚13007394275328 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚13674596936704 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 MADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚9940808 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6350926 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 697540218368 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 697371972096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚14496576159744 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚13702949280454 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚13007411314688 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 QEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚9959374 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚2757221 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6350924 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚14372155158016 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 697540218368 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚14372155158016 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚13674596936704 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 trainer/global_step‚499 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚16310300 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚993 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231084032 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 MADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚9959374 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚14371968908800 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚6461302 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216&Ttò>0×+ _wandbruntime‚377  _runtime‚377¢ÒßîSØÍ /loss/FRt3/assignment_loss‚6.388299731884972 2loss/SRqqt3/assignment_loss‚2.5602378043261442 1loss/FBt1/assignment_loss‚0.26535210377726426 0loss/SRqqt2/detection_loss‚3.053312045397858 1loss/SRqqt1/assignment_loss‚3.334538942575455 2loss/SRqqt4/assignment_loss‚3.1634620140338767 epoch‚0 1loss/FBt2/assignment_loss‚0.41663408785825595 0loss/FBt3/assignment_loss‚0.4035011177307906 .loss/FBt1/detection_loss‚5.830684004006563 _runtime‚ 377.747779698 /loss/FRt1/assignment_loss‚6.943061566352844 .loss/FRt2/detection_loss‚3.393961478861011 /loss/FRt2/assignment_loss‚6.682661348459672 .loss/FRt4/detection_loss‚3.366384446322918 1loss/SRqqt1/detection_loss‚4.0747353754937645 1loss/SRqqt4/detection_loss‚2.4445798826757175 trainer/global_step‚499 /loss/FRt1/detection_loss‚4.4384248852729815 % _timestamp‚1.7883957531110299e+09 _step‚39 1loss/SRqqt3/detection_loss‚1.8328572490726442 .loss/FBt3/detection_loss‚1.265052049408073 -loss/FBt4/detection_loss‚4.42474672794342 /loss/FRt4/assignment_loss‚6.950617303848267 2loss/SRqqt2/assignment_loss‚3.2682134681277804 1loss/FBt4/assignment_loss‚0.43610291948675045 /loss/FRt3/detection_loss‚2.5526240277025156 /loss/FBt2/detection_loss‚19.039115868508816 %loss/total_loss‚96.52916044912836ÌÑÑ–WÙÑ 1loss/SRqqt3/detection_loss‚1.8328572490726442 .loss/FBt3/detection_loss‚1.265052049408073 -loss/FBt4/detection_loss‚4.42474672794342 /loss/FRt4/assignment_loss‚6.950617303848267 2loss/SRqqt2/assignment_loss‚3.2682134681277804 1loss/FBt4/assignment_loss‚0.43610291948675045 /loss/FRt3/detection_loss‚2.5526240277025156 /loss/FBt2/detection_loss‚19.039115868508816 %loss/total_loss‚96.52916044912836 /loss/FRt3/assignment_loss‚6.388299731884972 2loss/SRqqt3/assignment_loss‚2.5602378043261442 1loss/FBt1/assignment_loss‚0.26535210377726426 0loss/SRqqt2/detection_loss‚3.053312045397858 1loss/SRqqt1/assignment_loss‚3.334538942575455 2loss/SRqqt4/assignment_loss‚3.1634620140338767 epoch‚0 1loss/FBt2/assignment_loss‚0.41663408785825595 0loss/FBt3/assignment_loss‚0.4035011177307906 .loss/FBt1/detection_loss‚5.830684004006563 _runtime‚ 377.747779698 /loss/FRt1/assignment_loss‚6.943061566352844 .loss/FRt2/detection_loss‚3.393961478861011 /loss/FRt2/assignment_loss‚6.682661348459672 .loss/FRt4/detection_loss‚3.366384446322918 1loss/SRqqt1/detection_loss‚4.0747353754937645 1loss/SRqqt4/detection_loss‚2.4445798826757175 trainer/global_step‚499 /loss/FRt1/detection_loss‚4.4384248852729815 % _timestamp‚1.7883957531110299e+09 _step‚39''"¡î‘Ú‚(:† õãÔñÿ­ gpu.0.gpu‚24.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚105.035' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.007 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ÁA PÛ‚(:Å õãÔ´²ï° disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 167.07421875 disk.drbd1716.out‚0 cpu‚25.106587956394446 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 643.34765625 disk.dm-1.in‚0 network.sent‚622053( disk./.usagePercent‚16.9014655980762& disk.nvme0c0n1.out‚9109.9482421875 disk.sda.out‚ 643.34765625 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 167.07421875$ disk.nvme0n1.out‚9109.9482421875 disk.dm-1.out‚ 643.34765625) proc.memory.availableMB‚ 2253.68359375 disk.sda2.in‚0 network.recv‚223238 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 167.07421875 disk.sda.in‚0 disk.sda2.out‚0 disk.drbd2265.out‚0! disk.dm-0.out‚9109.9482421875 disk.sda1.out‚0 disk.rbd1.out‚ 1.716796875 proc.memory.rssMB‚2588.375% disk./.usageGB‚251.84405136108398 disk.drbd2266.out‚0 disk.drbd1716.in‚0 disk.sda3.in‚0$ memory_percent‚86.24460697174072* proc.memory.percent‚15.798187255859375‰×Àw’Ü‚(:‡ „‚ãÔûÖ¡® gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚127.972' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.5944 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0öƒ]FÝ‚(:» „‚ãԀ汱% disk./.usageGB‚251.84405136108398 disk.rbd1.in‚0& disk.nvme0c0n1.out‚9110.1083984375 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0 proc.memory.rssMB‚2588.375 disk.sda1.out‚0 disk.sda2.in‚0* proc.memory.percent‚15.798187255859375( disk./.usagePercent‚16.9014655980762 disk.nvme0n1.in‚ 167.07421875 disk.sda.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.dm-1.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 167.07421875 cpu‚25.10455156604574 network.recv‚229203 disk.rbd1.out‚ 1.716796875" disk.nvme0c0n1.in‚ 167.07421875 disk.dm-1.out‚ 669.40625 disk.rbd0.out‚0 disk.sda3.out‚ 669.40625 disk.drbd1716.out‚0$ memory_percent‚86.24627590179443 network.sent‚672103$ disk.nvme0n1.out‚9110.1083984375 disk.sda.out‚ 669.40625 disk.drbd2080.out‚0! disk.dm-0.out‚9110.1083984375 proc.cpu.threads‚17 disk.sda3.in‚0) proc.memory.availableMB‚ 2253.41015625U/.0Þ+ _wandbruntime‚414  _runtime‚414ƒ×zÃß½ & lr-AdamW/pg1‚9.762419428320507e-05 & lr-AdamW/pg2‚9.762419428320507e-05 trainer/global_step‚549 $ _timestamp‚1.788395789863125e+09 _runtime‚ 414.499985341 _step‚40ÄjÄÁÇàÁ _runtime‚ 414.499985341 _step‚40 & lr-AdamW/pg1‚9.762419428320507e-05 & lr-AdamW/pg2‚9.762419428320507e-05 trainer/global_step‚549 $ _timestamp‚1.788395789863125e+09(éÏuu0á+ _wandbruntime‚414  _runtime‚414}PÊI±'â«O OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14278627249152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 779106688512 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6971217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10913723 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚545 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚15042247868574 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 765650156544 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚15776632201216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 765650156544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 765818389504 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚972 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10932263 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚15010795808768 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4065554 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 205060096 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17884938 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15776445965312 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17903480 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 763620619422 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 763455505566 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _runtime‚ 414.500961003 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15913310622720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 765818389504 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15042065715358 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚15776632201216 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10913723 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚15776445965312 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7090021 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚16692417311232 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6971217 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 229000192 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 779311748608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚16692646311424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚15010813811712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚547 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7883957898638706e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚993 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3024467 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚15913334562816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚15010795808768 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3025012 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2–^àã¯O UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚6971215 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 K?DeviceStatsMonitor.on_train_batch_start/active.large_poolA¤Ï:Õ%.freed‚6971215 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7090568 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚15010813811712 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚15913310622720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 765818389504 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚15042065715358 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚15776632201216 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚10913723 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚15776445965312 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚7090021 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚16692417311232 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚6971217 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 229000192 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 779311748608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚16692646311424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚15010813811712 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚547 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7883957898638706e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚993 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3024467 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚15913334562816 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚15010795808768 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3025012 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚14278627249152 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 779106688512 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚6971217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚10913723 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚545 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚15042247868574 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 765650156544 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚15776632201216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 765650156544 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 765818389504 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚972 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚10932263 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚15010795808768 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4065554 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 205060096 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚17884938 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚15776445965312 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚17903480 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 763620619422 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 763455505566 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _runtime‚ 414.500961003 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10933510 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 trainer/global_step‚549 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 767211540992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚15069580215662 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10952076 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15805294717440 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6983876 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚15038083176448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15069398062238 % _timestamp‚1.7883957905747902e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6983874 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3029988 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233181184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 764844670110 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10933510 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14304553392128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15805108468224 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17917384 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 765009784174 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6983874 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚15805294717440 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3030529 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚541 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17935952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚16722979044864 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10952076 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚15038083176448 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7102919 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 767211540992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚16722745863680 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17935952 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7103462 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6983876 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 767043294720 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚972 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚ 415.212356141 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚17935952 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚16722979044864 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚10952076 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚15038083176448 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7102919 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 767211540992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚16722745863680 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚17935952 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7103462 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚6983876 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 767043294720 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚993 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚972 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _runtime‚ 415.212356141 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚10933510 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 trainer/global_step‚549 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 767211540992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚15069580215662 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚10952076 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚15805294717440 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚6983876 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚15038083176448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚15069398062238 % _timestamp‚1.7883957905747902e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚6983874 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3029988 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 233181184 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 764844670110 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚10933510 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚14304553392128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚15805108468224 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚17917384 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 765009784174 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚6983874 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚15805294717440 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248*8B÷y0ç+ _wandbruntime‚415  _runtime‚415ü:kSèÍ _step‚43 /loss/FRt1/assignment_loss‚6.625305093567947 /loss/FRt3/assignment_loss‚5.809590831398964 2loss/SRqqt3/assignment_loss‚2.6245774739775163 trainer/global_step‚549 .loss/FRt2/assignment_loss‚6.43527307510376 1loss/SRqqt4/detection_loss‚2.1171275207675837 _runtime‚ 415.213354395 0loss/FBt4/assignment_loss‚0.4131183420023363 .loss/FRt4/detection_loss‚3.105158517482104 0loss/SRqqt1/detection_loss‚5.108601992018521 1loss/SRqqt3/detection_loss‚1.6156166985423654 2loss/SRqqt1/assignment_loss‚3.3726084008812904 0loss/FBt3/assignment_loss‚0.8166276670216273 .loss/FRt1/detection_loss‚3.118857354953373 /loss/FBt4/detection_loss‚4.1523363619214955 epoch‚0 0loss/FBt2/assignment_loss‚0.2763841749401763 1loss/SRqqt2/detection_loss‚2.0115657346008997 .loss/FBt1/detection_loss‚4.480033927980592 &loss/total_loss‚101.56590496146978 /loss/FRt2/detection_loss‚3.6951337920294867 1loss/SRqqt4/assignment_loss‚2.913333930571874 1loss/FBt1/assignment_loss‚0.28175526969511444 .loss/FRt3/detection_loss‚2.499054879648611 /loss/FBt2/detection_loss‚29.688221049308776 0loss/FRt4/assignment_loss‚5.7436464804190175 1loss/SRqqt2/assignment_loss‚2.861478094070677 /loss/FBt3/detection_loss‚1.8004982985656786 % _timestamp‚1.7883957905765622e+09[®^kWéÑ 0loss/FRt4/assignment_loss‚5.7436464804190175 1loss/SRqqt2/assignment_loss‚2.861478094070677 /loss/FBt3/detection_loss‚1.8004982985656786 % _timestamp‚1.7883957905765622e+09 _step‚43 /loss/FRt1/assignment_loss‚6.625305093567947 /loss/FRt3/assignment_loss‚5.809590831398964 2loss/SRqqt3/assignment_loss‚2.6245774739775163 trainer/global_step‚549 .loss/FRt2/assignment_loss‚6.43527307510376 1loss/SRqqt4/detection_loss‚2.1171275207675837 _runtime‚ 415.213354395 0loss/FBt4/assignment_loss‚0.4131183420023363 .loss/FRt4/detection_loss‚3.105158517482104 0loss/SRqqt1/detection_loss‚5.108601992018521 1loss/SRqqt3/detection_loss‚1.6156166985423654 2loss/SRqqt1/assignment_loss‚3.3726084008812904 0loss/FBt3/assignment_loss‚0.8166276670216273 .loss/FRt1/detection_loss‚3.118857354953373 /loss/FBt4/detection_loss‚4.1523363619214955 epoch‚0 0loss/FBt2/assignment_loss‚0.2763841749401763 1loss/SRqqt2/detection_loss‚2.0115657346008997 .loss/FBt1/detection_loss‚4.480033927980592 &loss/total_loss‚101.56590496146978 /loss/FRt2/detection_loss‚3.6951337920294867 1loss/SRqqt4/assignment_loss‚2.913333930571874 1loss/FBt1/assignment_loss‚0.28175526969511444 .loss/FRt3/detection_loss‚2.499054879648611 /loss/FBt2/detection_loss‚29.688221049308776+¨oCïoê‚(:’ “‚ãÔ‡—ì® gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.088' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.217599999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryEíu….rrors‚0# gpu.0.uncorrectedMemoryErrors‚0/;f0Së‚(:È “‚ãÔñéα disk.sda3.out‚ 714.87109375# memory_percent‚86.2494945526123( proc.memory.availableMB‚ 2252.8828125% disk./.usageGB‚251.84456634521484 disk.drbd1716.in‚0 disk.dm-1.in‚0 disk.nvme0n1.in‚ 167.07421875 cpu‚25.102684620314477 proc.cpu.threads‚17 network.sent‚678668 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda3.in‚0* proc.memory.percent‚15.798187255859375 disk.rbd0.in‚0 disk.dm-0.in‚ 167.07421875 disk.drbd2080.out‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚9110.43115234375% disk.nvme0n1.out‚9110.43115234375 proc.memory.rssMB‚2588.375 disk.rbd1.out‚ 1.7490234375 network.recv‚234441 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0" disk.dm-0.out‚9110.43115234375 disk.rbd1.in‚0 disk.sda.in‚0 disk.sda.out‚ 714.87109375 disk.drbd2266.in‚0) disk./.usagePercent‚16.90150015909332 disk.dm-1.out‚ 714.87109375 disk.drbd2266.out‚0 disk.drbd2080.in‚0" disk.nvme0c0n1.in‚ 167.07421875 disk.sda2.in‚0 disk.sda1.out‚0²‘`kOì‚(:Ä ¢‚ãÔ‡‰±* proc.memory.percent‚15.798211097717285 network.sent‚727985 disk.rbd0.in‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 167.07421875% disk.nvme0n1.out‚9110.70458984375% disk./.usageGB‚251.84450912475586 disk.sda.out‚ 750.96875 disk.drbd2266.out‚0 cpu‚25.10103012271287 network.recv‚239515* disk./.usagePercent‚16.901496318980307 disk.sda.in‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 2253.296875 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.dm-0.in‚ 167.07421875$ memory_percent‚86.24696731567383 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 1.7490234375# proc.memory.rssMB‚ 2588.37890625 disk.drbd1716.in‚0" disk.dm-0.out‚9110.70458984375 disk.rbd1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 750.96875 disk.drbd2080.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 750.96875' disk.nvme0c0n1.out‚9110.70458984375 disk.nvme0n1.in‚ 167.07421875 disk.sda2.in‚0 disk.drbd2265.out‚0ŽðÒ‘í‚(:† ¢‚ãÔÓ¹ø¸ gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚113.775' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.755 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚16374282048512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19529132 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 835321479168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 _step‚45 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚17358981974016 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17209585524736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11906425 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚599 _runtime‚ 452.370853211 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11924965 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚17209585524736 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚17209771760640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚972 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15575769326592 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7604167 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16374264045568 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3298694 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚16408696650566 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7883958277336636e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚993 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7604165 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7604167 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11924965 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚16374264045568 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7733631 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3299225 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚18208920251392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4434404 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 833092437830 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11906425 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 835489712128 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 849962217472 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16408878803782 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233194496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 850171471872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630v¤íøµ'õ¯O PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚11924965 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚17209585524736 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚17209771760640 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚972 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚15575769326592 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 SGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚7604167 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚16374264045568 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚3298694 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚16408696650566 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7883958277336636e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 G?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚993 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚7604165 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚7604167 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚11924965 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚16374264045568 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚7733631 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3299225 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚18208920251392 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚4434404 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 833092437830 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚11906425 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 835489712128 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 849962217472 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚16408878803782 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233194496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 850171471872 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚18209153445888 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚17358958033920 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚16374282048512 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚19529132 VEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 835321479168 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 _step‚45 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚17358981974016 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚17209585524736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚11906425 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚599 _runtime‚ 452.370853211-3`•o0ö+ _wandbruntime‚453  _runtime‚453àãV Ï÷µM QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 _runtime‚ 453.074644886 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚16401533410304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11944778 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 851371451904 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 836882863616 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7616824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚17387870982144 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16436211150870 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19561604 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _step‚46 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚547 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7616826 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4441783 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚17238434276864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 239472640 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7616824 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19561604 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚16436028997446 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 836714617344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚17238248027648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7746404 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚17238434276864 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7616826 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 836882863616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16401533410304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 % _timestamp‚1.7883958284372888e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11926212 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11944778 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚16401551413248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚993 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3304621 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚549 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3304074 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚18239242434048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 851586984448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 215532544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15601712508928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19543036 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7745855 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚18239481906688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11926212q Äò¿&ø¹M QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚7616826 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4441783 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚17238434276864 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 239472640 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚7616824 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚19561604 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚16436028997446 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 836714617344 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚17238248027648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚7746404 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚17238434276864 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚7616826 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 XGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 836882863616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚16401533410304 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 % _timestamp‚1.7883958284372888e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚11926212 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚11944778 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚16401551413248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 E=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚993 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3304621 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚549 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚3304074 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚18239242434048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 851586984448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 215532544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚15601712508928 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚19543036 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚7745855 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚18239481906688 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚11926212 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 _runtime‚ 453.074644886 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚16401533410304 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚11944778 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 851371451904 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 836882863616 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚7616824 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚17387870982144 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚16436211150870 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚19561604 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _step‚46 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚547 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0.ã¡W0ù+ _wandbruntime‚453  _runtime‚453‘z+´QúË epoch‚0 /loss/FRt1/assignment_loss‚6.553572251246526 /loss/FBt4/detection_loss‚3.5705690320581196 1loss/SRqqt4/detection_loss‚2.4195451521272844 1loss/SRqqt1/assignment_loss‚3.282967300126047 2loss/SRqqt2/assignment_loss‚2.9403588947577353 2loss/SRqqt3/assignment_loss‚2.3953243827415727 .loss/FRt1/detection_loss‚3.934594823763921 _runtime‚ 453.075806589 _step‚47 0loss/FBt2/assignment_loss‚0.8713697592417399 1loss/SRqqt2/detection_loss‚1.4377094244141166 0loss/FBt4/assignment_loss‚0.3052309337508632 2loss/SRqqt4/assignment_loss‚2.8726299442350864 0loss/FBt3/assignment_loss‚0.6972502639982849 .loss/FRt2/detection_loss‚3.777122755383336 0loss/SRqqt1/detection_loss‚4.878220395608381 1loss/SRqqt3/detection_loss‚1.9889614599811385 .loss/FBt3/detection_loss‚4.011386929389923 %loss/total_loss‚91.39785624036222 trainer/global_step‚599 0loss/FRt4/assignment_loss‚5.3784070412317915 0loss/FBt1/assignment_loss‚0.5428448114668883 /loss/FRt3/detection_loss‚2.7279392342295563 -loss/FRt4/detection_loss‚3.56224302140375 /loss/FRt2/assignment_loss‚5.945424279501272 /loss/FRt3/assignment_loss‚5.324875404960231 .loss/FBt1/detection_loss‚5.590824770075934 /loss/FBt2/detection_loss‚16.388483974668716 % _timestamp‚1.7883958284390416e+09Š:IyUûÏ 0loss/FBt2/assignment_loss‚0.8713697592417399 1loss/SRqqt2/detection_loss‚1.4377094244141166 0loss/FBt4/assignment_loss‚0.3052309337508632 2loss/SRqqt4/assignment_loss‚2.8726299442350864 0loss/FBt3/assignment_loss‚0.6972502639982849 .loss/FRt2/detection_loss‚3.777122755383336 0loss/SRqqt1/detection_loss‚4.878220395608381 1loss/SRqqt3/detection_loss‚1.9889614599811385 .loss/FBt3/detection_loss‚4.011386929389923 %loss/total_loss‚91.39785624036222 trainer/global_step‚599 0loss/FRt4/assignment_loss‚5.3784070412317915 0loss/FBt1/assignment_loss‚0.5428448114668883 /loss/FRt3/detection_loss‚2.7279392342295563 -loss/FRt4/detection_loss‚3.56224302140375 /loss/FRt2/assignment_loss‚5.945424279501272 /loss/FRt3/assignment_loss‚5.324875404960231 .loss/FBt1/detection_loss‚5.590824770075934 /loss/FBt2/detection_loss‚16.388483974668716 % _timestamp‚1.7883958284390416e+09 epoch‚0 /loss/FRt1/assignment_loss‚6.553572251246526 /loss/FBt4/detection_loss‚3.5705690320581196 1loss/SRqqt4/detection_loss‚2.4195451521272844 1loss/SRqqt1/assignment_loss‚3.282967300126047 2loss/SRqqt2/assignment_loss‚2.9403588947577353 2loss/SRqqt3/assignment_loss‚2.3953243827415727 .loss/FRt1/detection_loss‚3.934594823763921 _runtime‚ 453.075806589 _step‚47/‘bž’ü‚(:‡ À‚ãÔ󀚮 gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚119.973' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9946 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n”Oý‚(:Ä À‚ãÔÀ‚Ÿ± disk.sda.out‚ 815.46875 disk.drbd1716.in‚0 disk.dm-1.out‚ 815.46875$ memory_percent‚86.24932765960693* proc.memory.percent‚15.798211097717285 disk.dm-0.in‚ 167.07421875 disk.sda1.in‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 167.07421875# proc.memory.rssMB‚ 2588.37890625 cpu‚25.096866078617328 network.recv‚248551% disk./.usageGB‚251.84450912475586 disk.sda1.out‚0& disk.nvme0c0n1.out‚9111.2978515625 disk.sda2.out‚0 disk.dm-1.in‚0 proc.cpu.threads‚17 disk.drbd2265.in‚0$ disk.nvme0n1.out‚9111.2978515625) proc.memory.availableMB‚ 2252.91015625 network.sent‚740597 disk.drbd2266.in‚0 disk.sda.in‚0! disk.dm-0.out‚9111.2978515625 disk.rbd1.out‚ 1.9052734375 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0 disk.sda3.out‚ 815.46875 disk.nvme0n1.in‚ 167.07421875* disk./.usagePercent‚16.901496318980307 disk.drbd2266.out‚0 disk.drbd2080.out‚0$€i%’þ‚(:‡ Ï‚ãÔÔ°ü¯ gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.088' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6176 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á©äb[ÿ‚(:Ð Ï‚ãÔ—¥¾± disk.sda.in‚0 disk.drbd1716.in‚0 disk.sda3.out‚ 874.26953125) proc.memory.availableMB‚ 2253.17578125 disk.drbd2266.out‚0 disk.dm-1.in‚0 disk.dm-1.out‚ 874.26953125 disk.rbd0.in‚0* proc.memory.percent‚15.798211097717285 network.recv‚254580 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚9111.80322265625 disk.sda.out‚ 874.26953125 disk.drbd2080.in‚0 disk.dm-0.in‚ 167.07421875" disk.nvme0c0n1.in‚ 167.07421875% disk.nvme0n1.out‚9111.80322265625 disk.sda1.out‚0 disk.sda3.in‚0$ memory_percent‚86.24770641326904 cpu‚25.095943340299172 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0" disk.dm-0.out‚9111.80322265625 proc.cpu.threads‚17 network.sent‚790854# proc.memory.rssMB‚ 2588.37890625* disk./.usagePercent‚16.901496318980307% disk./.usageGB‚251.84450912475586 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 1.9052734375 disk.nvme0n1.in‚ 167.07421875 cÛÅ0€+ _wandbruntime‚490  _runtime‚490cýÆÀ ' lr-AdamW/pg1‚0.00011540637903424423 ' lr-AdamW/pg2‚0.00011540637903424423 trainer/global_step‚649 % _timestamp‚1.7883958654799936e+09 _runtime‚ 490.116958981 _step‚48>Ê‚Ä % _timestamp‚1.7883958654799936e+09 _runtime‚ 490.116958981 _step‚48 ' lr-AdamW/pg1‚0.00011540637903424423 ' lr-AdamW/pg2‚0.00011540637903424423 trainer/global_step‚6490¤~P~0ƒ+ _wandbruntime‚490  _runtime‚490D@ò²'„¬O G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3575992 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18642725084160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _step‚49 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚18642911320064 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚17737750285312 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202962944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚4803256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚649 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19725636160512 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16872945482752 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7883958654808187e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 905161034752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 905161034752 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8237115 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16872928443392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8237115 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚17737732282368 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226903040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚17775509738990 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 902564256238 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚18642725084160 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8378707 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18804605445120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 902399142382 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21136242 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚18642911320064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1002 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18804629385216 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12899127 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12917667 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚17775327585774 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 920803812352 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 _runtime‚ 490.117914384Ì æ¾¦…°O IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚12917667 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚17775327585774 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 920803812352 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 _runtime‚ 490.117914384 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚3575992 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚18642725084160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _step‚49 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚18642911320064 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚17737750285312 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202962944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.la.rge_pool.allocated‚4803256 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚649 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚19725636160512 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚16872945482752 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7883958654808187e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 905161034752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 905161034752 OCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚8237115 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚16872928443392 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8237115 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚17737732282368 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226903040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚17775509738990 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 902564256238 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚18642725084160 L@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚8378707 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚18804605445120 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 902399142382 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚21136242 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚18642911320064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1002 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚18804629385216 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚12899127 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 47572377091^¼i0†+ _wandbruntime‚490  _runtime‚490ÄÉÅ`¼&‡¶M ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8391759 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16898888665088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3581660 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 213435392 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚981 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 903788306926 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚18671387587072 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21187256 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚ 490.832491645 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8392293 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21168688 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8249776 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚50 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚17765019650048 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 906554186240 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1002 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4810633 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19755732697600 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 922214304256 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 903953420990 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8249774 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚18671573836288 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 trainer/global_step‚649 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12918914 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚17802842086078 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21187256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚17765019650048 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19755970073088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚534 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18833542333440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 922427739648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8249774 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18833518393344 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚17802659932654 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0sÄU¤À&ˆºM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚18671387587072 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚21187256 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚ 490.832491645 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚8392293 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚21168688 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8249776 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _step‚50 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚17765019650048 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 906554186240 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1002 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚4810633 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚19755732697600 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 922214304256 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 [JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 903953420990 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚8249774 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚18671573836288 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 trainer/global_step‚649 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚12918914 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚17802842086078 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚21187256 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚17765019650048 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚19755970073088 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚534 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚18833542333440 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 922427739648 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8249774 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚18833518393344 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚17802659932654 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚8391759 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚16898888665088 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3581660 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 213435392 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚981 WFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9037883069262Ä?ˆQ0‰+ _wandbruntime‚490  _runtime‚490ÊYžÐUŠÏ 1loss/FBt1/assignment_loss‚0.37605916443280873 /loss/FBt3/detection_loss‚3.3944064358967565 2loss/SRqqt2/assignment_loss‚2.9059188295813168 0loss/FBt2/assignment_loss‚0.6180384857968116 0loss/FBt4/assignment_loss‚0.2496100850133059 1loss/SRqqt2/detection_loss‚2.5367851602096185 epoch‚0 0loss/FBt3/assignment_loss‚0.4771817368303385 .loss/FRt1/detection_loss‚6.164386509095921 0loss/SRqqt4/detection_loss‚1.399149329454191 _step‚51 /loss/FRt2/assignment_loss‚5.680424435194149 /loss/FRt4/assignment_loss‚4.926185338585465 /loss/FBt2/detection_loss‚23.202120065689087 trainer/global_step‚649 % _timestamp‚1.7883958661969426e+09 /loss/FRt1/assignment_loss‚6.536599888521082 1loss/SRqqt4/assignment_loss‚2.658146908253799 /loss/FRt2/detection_loss‚3.6826311426107274 /loss/FRt4/detection_loss‚2.9151113952199617 /loss/FBt1/detection_loss‚10.081170656283696 /loss/FBt4/detection_loss‚2.1730958304501007 /loss/FRt3/detection_loss‚3.0775146274005665 0loss/SRqqt3/detection_loss‚4.574372070954408 &loss/total_loss‚104.65912897096874 _runtime‚ 490.833726025 /loss/FRt3/assignment_loss‚5.522955062342625 1loss/SRqqt1/assignment_loss‚3.487398902575175 2loss/SRqqt3/assignment_loss‚2.9216461877028146 1loss/SRqqt1/detection_loss‚5.0982207228740055raE,Y‹Ó 1loss/FBt1/assignment_loss‚0.37605916443280873 /loss/FBt3/detection_loss‚3.3944064358967565 2loss/SRqqt2/assignment_loss‚2.9059188295813168 0loss/FBt2/assignment_loss‚0.6180384857968116 0loss/FBt4/assignment_loss‚0.2496100850133059 1loss/SRqqt2/detection_loss‚2.5367851602096185 epoch‚0 0loss/FBt3/assignment_loss‚0.4771817368303385 .loss/FRt1/detection_loss‚6.164386509095921 0loss/SRqqt4/detection_loss‚1.399149329454191 _step‚51 /loss/FRt2/assignment_loss‚5.680424435194149 /loss/FRt4/assignment_loss‚4.926185338585465 /loss/FBt2/detection_loss‚23.202120065689087 trainer/global_step‚649 % _timestamp‚1.7883958661969426e+09 /loss/FRt1/assignment_loss‚6.536599888521082 1loss/SRqqt4/assignment_loss‚2.658146908253799 /loss/FRt2/detection_loss‚3.6826311426107274 /loss/FRt4/detection_loss‚2.9151113952199617 /loss/FBt1/detection_loss‚10.081170656283696 /loss/FBt4/detection_loss‚2.1730958304501007 /loss/FRt3/detection_loss‚3.0775146274005665 0loss/SRqqt3/detection_loss‚4.574372070954408 &loss/total_loss‚104.65912897096874 _runtime‚ 490.833726025 /loss/FRt3/assignment_loss‚5.522955062342625 1loss/SRqqt1/assignment_loss‚3.487398902575175 2loss/SRqqt3/assignment_loss‚2.9216461877028146 1loss/SRqqt1/detection_loss‚5.09822072287400553ÁçϽ’Œ‚(:‡ Þ‚ãÔæ´š® gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚127.528' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.5056 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0yzã¾W‚(:Ì Þ‚ãÔÙ¿ë± network.recv‚258723* disk./.usagePercent‚16.901496318980307 disk.drbd2080.in‚0 disk.sda3.in‚0% disk.nvme0n1.out‚9112.00634765625 disk.sda2.in‚0# memory_percent‚86.2480878829956% disk./.usageGB‚251.84450912475586 disk.dm-1.in‚0 disk.dm-0.in‚ 167.07421875 disk.rbd1.in‚0 cpu‚25.094101514063485" disk.dm-0.out‚9112.00634765625 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 167.07421875 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.drbd2265.out‚0' disk.nvme0c0n1.out‚9112.00634765625 disk.nvme0n1.in‚ 167.07421875 disk.drbd1716.out‚0 disk.dm-1.out‚ 902.89453125 disk.sda1.out‚0 disk.sda2.out‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 2253.11328125 network.sent‚796591 disk.drbd2266.out‚0 disk.sda3.out‚ 902.89453125 disk.rbd1.out‚ 2.0234375 disk.sda.out‚ 902.89453125* proc.memory.percent‚15.798211097717285 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda.in‚0 disk.drbd1716.in‚0# proc.memory.rssMB‚ 2588.37890625ù•jÀ’Ž‚(:‡ í‚ãÔìÈÁ® gpu.0.gpu‚27.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚113.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7068 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u‚úÂT‚(:É í‚ãÔޗ̱ network.recv‚264477 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda3.in‚0 disk.rbd1.out‚ 2.1171875' disk.nvme0c0n1.out‚9112.41748046875 disk.nvme0n1.in‚ 167.07421875 disk.sda.in‚0$ memory_percent‚86.25054359436035* proc.memory.percent‚15.798211097717285% disk./.usageGB‚251.84417343139648 disk.sda2.in‚0 disk.dm-1.out‚ 928.5234375 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 167.07421875* disk./.usagePercent‚16.901473790317294 disk.drbd1716.in‚0 disk.dm-1.in‚0 disk.drbd2266.in‚0 disk.drbd2265.in‚0" disk.dm-0.out‚9112.41748046875% disk.nvme0n1.out‚9112.41748046875( proc.memory.availableMB‚ 2252.7109375 network.sent‚846689 disk.sda.out‚ 928.5234375 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 167.07421875# proc.memory.rssMB‚ 2588.37890625 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda3.out‚ 928.5234375 cpu‚25.092888363399005 disk.drbd1716.out‚0 disk.rbd1.in‚0ëݺ³’‚(:‡ ü‚ãÔçô® gpu.0.gpu‚29.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚106.798' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3596 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'߾𪑂(:â ü‚ãÔôêß° disk.drbd2080.out‚0 disk.rbd0.in‚0) proc.memory.availableMB‚ 2252.69921875* proc.memory.percent‚15.798211097717285* disk./.usagePercent‚16.901478142445377 disk.dm-1.out‚ 971.7890625 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda3.out‚ 971.7890625" disk.nvme0c0n1.in‚ 167.07421875 disk.sda2.in‚0 network.sent‚852910 disk.drbd2265.in‚0 disk.8ˆY¡Ãdrbd1716.out‚0 disk.dm-0.in‚ 167.07421875 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0" disk.dm-0.out‚9112.58544921875' disk.nvme0c0n1.out‚9112.58544921875 disk.sda3.in‚ 0.00390625 disk.rbd1.in‚0 disk.nvme0n1.in‚ 167.07421875% disk.nvme0n1.out‚9112.58544921875 network.recv‚269182" disk./.usageGB‚251.84423828125 disk.drbd2265.out‚0 disk.sda.in‚ 0.00390625 disk.sda2.out‚0$ memory_percent‚86.25061511993408 cpu‚25.091799547661925 proc.cpu.threads‚17 disk.dm-1.in‚ 0.00390625 disk.rbd0.out‚0 disk.rbd1.out‚ 2.1796875 disk.sda.out‚ 971.7890625# proc.memory.rssMB‚ 2588.37890625&iÈ­0’+ _wandbruntime‚527  _runtime‚527ÞÊÚÆ“À ' lr-AdamW/pg1‚0.00012429747140976384 ' lr-AdamW/pg2‚0.00012429747140976384 trainer/global_step‚699 % _timestamp‚1.7883959032639616e+09 _runtime‚ 527.900855538 _step‚52³èJÊ”Ä ' lr-AdamW/pg1‚0.00012429747140976384 ' lr-AdamW/pg2‚0.00012429747140976384 trainer/global_step‚699 % _timestamp‚1.7883959032639616e+09 _runtime‚ 527.900855538 _step‚524\_Öh0•+ _wandbruntime‚527  _runtime‚527~›±'–«O ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚53 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚19101218522112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 974832357376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8870065 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20075864643584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 972036074646 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8870067 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 991668340224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 971870960790 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 974832357376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9024817 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13891829 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21241921196544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13910369 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚20250252856320 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 991879691776 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚18170104599552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13891829 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚20076050879488 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 974664124416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22780436 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚21242156488192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚20076050879488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235291648 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚19101218522112 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚19142140674198 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1002 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5172104 _runtime‚ 527.90182557 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚19101200519168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1002 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚5172104 _runtime‚ 527.90182557 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚19101200519168 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚53 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚19101218522112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZIDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 974832357376 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚8870065 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚20075864643584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 ]LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 972036074646 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 OCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚8870067 ^MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 991668340224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 971870960790 ]LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 974832357376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 PDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚9024817 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚13891829 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚21241921196544 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚13910369 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚20250252856320 bQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 991879691776 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚18170104599552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚13891829 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚20076050879488 YHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 974664124416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚22780436 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚21242156488192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚20076050879488 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 235291648 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚19101218522112 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚19142140674198 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚981 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 993089826304 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8882724 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 993292775936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 trainer/global_step‚699 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13911616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226889728 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3858233 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20279165804544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚18196030742528 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5179483 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 976057262592 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚19128469883904 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1002 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22812908 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚19128469883904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9037188 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21272482520576 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 976057262592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚19128487886848 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202949632 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22812908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚19128487886848 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20104713395712 _runtime‚ 528.605596141 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13930182 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚18196047781888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199Øâþ·š¸M RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 [JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 976225508864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚22812908 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚19128487886848 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚20104713395712 _runtime‚ 528.605596141 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚13930182 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚18196047781888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytesÆ€Ì/".large_pool.allocated‚20279189744640 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _step‚54 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚22794340 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚8882726 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚981 \KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 993089826304 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚8882724 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 `ODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 993292775936 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 trainer/global_step‚699 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚13911616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226889728 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚3858233 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚20279165804544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚18196030742528 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5179483 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 976057262592 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚19128469883904 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1002 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚22812908 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚19128469883904 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9037188 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚21272482520576 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 976057262592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚19128487886848 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202949632 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚20508589456190 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9503015 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14884531 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14903071 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1002 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1041342779198 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1044335447040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚21695924207616 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7883959410064628e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1044335447040 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚24406088 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚20464668755968 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚981 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚21509004203008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _runtime‚ 565.643613145 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚20464686758912 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1044503680000 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚21509004203008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _runtime‚ 565.643613145 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚20464686758912 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1044503680000 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚20508589456190 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 K?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚9503015 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚14884531 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚14903071 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1002 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1041342779198 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1044335447040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚269êMk0¨+ _wandbruntime‚566  _runtime‚566ƒDTÄ&©¾M G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24438560 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 _step‚58 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1042731943742 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9683359 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚21537852955136 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231084032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24419992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚19493206898688 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4135026 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚981 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9515674 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 207143936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9682840 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1002 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4134509 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚ 566.351293545 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚20491956123648 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5548333 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚21724813215744 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14922884 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9515676 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14922884 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14904318 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1045728585216 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24419992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚19493189859328 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚20491938120704 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9515674 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14904318 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1045896831488 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5548331 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24438560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚22788747641856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1045728585216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7883959417138317e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9515676 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚20536103956494 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064„×wÈ&ªÂM I=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚9515674 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚14904318 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1045896831488 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚5548331 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚24438560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚22788747641856 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1045728585216 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7883959417138317e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚9515676 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚20536103956494 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚24438560 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 _step‚58 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1042731943742 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 NBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚9683359 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚21537852955136 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231084032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚24419992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚19493206898688 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4135026 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚981 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 MADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚9515674 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 207143936 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚9682840 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1002 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4134509 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚ 566.351293545 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚20491956123648 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚5548333 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚21724813215744 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚14922884 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚9515676 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚14922884 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚14904318 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1045728585216 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚24419992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚19493189859328 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚20491938120704 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460:EP‘Ð0«+ _wandbruntime‚566  _runtime‚566ºÒÚMT¬Î 1loss/SRqqt4/detection_loss‚1.8946404323555726 /loss/FBt2/detection_loss‚14.275177975495657 /loss/FBt4/detection_loss‚1.8595260659893673 _runtime‚ 566.352648233 /loss/FRt3/assignment_loss‚5.745999000966549 /loss/FRt4/detection_loss‚3.2640628966879337 2loss/SRqqt1/assignment_loss‚2.7953553010116923 .loss/FRt1/detection_loss‚2.514347791671753 /loss/FRt3/detection_loss‚2.4545479917433113 0loss/SRqqt3/detection_loss‚4.222637745312282 .loss/FBt1/detection_loss‚5.448016174710714 2loss/SRqqt4/assignment_loss‚2.5436119101941586 1loss/FBt3/assignment_loss‚0.43188141086804016 /loss/FBt3/detection_loss‚3.9727138181056483 epoch‚0 % _timestamp‚1.7883959417158852e+09 /loss/FRt4/assignment_loss‚4.456879027346347 1loss/FBt1/assignment_loss‚0.34973113347391166 0loss/FBt2/assignment_loss‚0.6931477321518792 /loss/FRt2/detection_loss‚3.6038341231124345 _step‚59 1loss/SRqqt2/assignment_loss‚2.850512321033175 /loss/SRqqt1/detection_loss‚6.48613366349177 1loss/SRqqt2/detection_loss‚2.0929717722511483 %loss/total_loss‚87.31444291454898 /loss/FRt1/assignment_loss‚6.122731032194914 0loss/FBt4/assignment_loss‚0.3213562320336003 trainer/global_step‚749 /loss/FRt2/assignment_loss‚5.706460586813993 2loss/SRqqt3/assignment_loss‚3.2081667755331313ÿ¦+ÚX­Ò 0loss/FBt2/assignment_loss‚0.6931477321518792 /loss/FRt2/detection_loss‚3.6038341231124345 _step‚59 1loss/SRqqt2/assignment_loss‚2.850512321033175 /loss/SRqqt1/detection_loss‚6.48613366349177 1loss/SRqqt2/detection_loss‚2.0929717722511483 %loss/total_loss‚87.31444291454898 /loss/FRt1/assignment_loss‚6.122731032194914 0loss/FBt4/assignment_loss‚0.3213562320336003 trainer/global_step‚749 /loss/FRt2/assignment_loss‚5.706460586813993 2loss/SRqqt3/assignment_loss‚3.2081667755331313 1loss/SRqqt4/detection_loss‚1.8946404323555726 /loss/FBt2/detection_loss‚14.275177975495657 /loss/FBt4/detection_loss‚1.8595260659893673 _runtime‚ 566.352648233 /loss/FRt3/assignment_loss‚5.745999000966549 /loss/FRt4/detection_loss‚3.2640628966879337 2loss/SRqqt1/assignment_loss‚2.7953553010116923 .loss/FRt1/detection_loss‚2.514347791671753 /loss/FRt3/detection_loss‚2.4545479917433113 0loss/SRqqt3/detection_loss‚4.222637745312282 .loss/FBt1/detection_loss‚5.448016174710714 2loss/SRqqt4/assignment_loss‚2.5436119101941586 1loss/FBt3/assignment_loss‚0.43188141086804016 /loss/FBt3/detection_loss‚3.9727138181056483 epoch‚0 % _timestamp‚1.7883959417158852e+09 /loss/FRt4/assignment_loss‚4.456879027346347 1loss/FBt1/assignment_loss‚0.34973113347391166;=îw®‚(:’ ©ƒãÔ¯¸® gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚111.353' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.270599999999998 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:Ç^gm¯‚(:â ©ƒãÔÍÆú± disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 167.07421875 disk.nvme0n1.in‚ 167.07421875) proc.memory.availableMB‚ 2252.48828125 disk.sda2.out‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚9113.60302734375$ memory_percent‚86.25190258026123% disk./.usageGB‚251.84536361694336 disk.sda2.in‚0 disk.drbd2080.in‚0" disk.dm-0.out‚9113.60302734375 disk.sda3.out‚ 1056.546875 disk.sda.in‚ 0.0078125 disk.sda3.in‚ 0.0078125# proc.memory.rssMB‚ 2588.37890625* proc.memory.percent‚15.798211097717285 cpu‚25.0885312065159 disk.sda.out‚ 1056.546875 disk.dm-1.in‚ 0.0078125 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 167.07421875) disk./.usagePercent‚16.90155366466798 disk.rbd1.out‚ 2.3916015625 proc.cpu.threads‚17 network.sent‚915357 disk.dm-1.out‚ 1056.546875 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚9113.60302734375 network.recv‚283989 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.drbd2265.in‚0ê}è’°‚(:‡ ¸ƒãÔ´µ—® gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚131.601' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.3202 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0hw`”t±‚(:é ¸ƒãÔä‚—±* disk./.usagePercent‚16.901552384630307 disk.sda3.out‚ 1108.08984375$ memory_percent‚86.25273704528809( proc.memory.availableMB‚ 2252.3515625 disk.drbd2265.out‚0 disk.dm-0.in‚ 167.07421875 disk.sda2.in‚0 disk.dm-1.in‚ 0.0078125 disk.drbd2080.out‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚9113.80224609375" disk.nvme0c0n1.in‚ 167.07421875 disk.sda.out‚ 1108.08984375" proc.memory.rssMB‚ 2588.3828125 cpu‚25.087240884767308 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 167.07421875 disk.drbd1716.out‚0 disk.dm-1.out‚ 1108.08984375 proc.cpu.threads‚17 network.sent‚965402" disk.dm-0.out‚9113.80224609375% disk.nvme0n1.out‚9113.80224609375 disk.drbd2266.in‚0* proc.memory.percent‚15.798234939575195 disk.drbd1716.in‚0 network.recv‚289895 disk.sda3.in‚ 0.0078125 disk.rbd1.in‚0 disk.rbd1.out‚ 2.4541015625 disk.sda.in‚ 0.0078125% disk./.usageGB‚251.84534454345703 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd2266.out‚0ÙrIy’²‚(:‡ ǃãԆР® gpu.0.gpu‚34.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚125.106' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.0212 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bªƒ4s³‚(:è ǃãԧ봰 disk.sda1.in‚0' disk.nvme0c0n1.out‚9114.71923828125# proc.memory.rssMB‚ 2588.38671875* proc.memory.percent‚15.798258781433105 cpu‚25.086011104007913* disk./.usagePercent‚16.901564160976882 disk.drbd1716.out‚0$ memory_percent‚86.25266551971436 proc.cpu.threads‚17 network.sent‚972344 network.recv‚295560 disk.dm-1.out‚ 1137.8046875 disk.drbd2266.in‚0 disk.sda3.in‚ 0.0078125% disk.nvme0n1.out‚9114.71923828125 disk.sda.in‚ 0.0078125 disk.drbd2080.in‚0 disk.drbd2080.out‚0" disk.dm-0.out‚9114.71923828125) proc.memory.availableMB‚ 2252.36328125% disk./.usageGB‚251.84552001953125 disk.drbd2266.out‚0 disk.dm-0.in‚ 167.07421875 disk.sda1.out‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 167.07421875 disk.nvme0n1.in‚ 167.07421875 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.0078125 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.sda3.out‚ 1137.8046875 disk.sda.out‚ 1137.8046875 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.4541015625\çA,0´+ _wandbruntime‚603  _runtime‚6032$_õ¼ % lr-AdamW/pg1‚0.000142079656160803 % lr-AdamW/pg2‚0.000142079656160803 trainer/global_step‚799 % _timestamp‚1.7883959785379531e+09 _runtime‚ 603.174808424 _step‚60x¹ézƶÀ % _timestamp‚1.7883959785379531e+09 _runtime‚ 603.174808424 _step‚60 % lr-AdamW/pg1‚0.000142079656160803 % lr-AdamW/pg2‚0.000142079656160803 trainer/global_step‚799<óúÊ—0·+ _wandbruntime‚603  _runtime‚603‚ýJÏ.¸ºO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚10135967 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚15895773 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚21828154995712 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚575 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚23141547678720 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚15877233 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚61 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 205060096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 trainer/global_step‚799 _runtime‚ 603.175826867 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segme0q­’nts.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10135967 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4406253 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1110979711462 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1114006769664 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚21828154995712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1114175002624 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚21875220391398 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1133316482048 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10135965 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10135965 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚20764405793792 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22942143762432 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22942143762432 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4405680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 229000192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10315484 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1133521542144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚20764422833152 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1114006769664 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚5909806 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚23141571618816 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚24275093160960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚21875402544614 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚21875402544614 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚10135967 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4406253 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1110979711462 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1114006769664 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚21828154995712 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1114175002624 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚21875220391398 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1133316482048 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚10135965 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10135965 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚20764405793792 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚22942143762432 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚22942143762432 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚4405680 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 229000192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚10315484 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1133521542144 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚20764422833152 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856=é }0º+ _wandbruntime‚603  _runtime‚6031„¸É&»ÃM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15897020 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚20790348976128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 $ _timestamp‚1.788395979246602e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚21855406357504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26064212 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1115399907840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4411152 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚23170460626944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22970992514560 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚21855424360448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1112368876214 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚24305196769792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202949632 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226889728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1112203762150 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15897020 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10148624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10328861 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚21855424360448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10148626 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1115568154112 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20790366015488 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26045644 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1115568154112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15915586 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1134736142848 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10328333 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23170484567040 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚62 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26064212 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1115399907840 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22970806265344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10148626 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚24305423659520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22970992514560 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416'“鎼ÇM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10328333 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚23170484567040 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚62 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚26064212 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1115399907840 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚22970806265344 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10148626 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚24305423659520 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚22970992514560 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚15897020 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.currÓS¨Œ?ent‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚20790348976128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 $ _timestamp‚1.788395979246602e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚21855406357504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚26064212 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1115399907840 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4411152 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚23170460626944 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚22970992514560 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚21855424360448 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1112368876214 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚24305196769792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202949632 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226889728 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1112203762150 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚15897020 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10148624 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10328861 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚21855424360448 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10148626 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1115568154112 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚20790366015488 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚26045644 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1115568154112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚15915586 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1134736142848>“»¸0½+ _wandbruntime‚603  _runtime‚603×Yl=Q¾Ë $ _timestamp‚1.788395979248281e+09 2loss/SRqqt4/assignment_loss‚2.6868113449641635 0loss/FBt1/assignment_loss‚0.5079044644621068 /loss/FBt4/detection_loss‚2.5807563456948155 &loss/total_loss‚105.26321017409089 /loss/FRt3/assignment_loss‚5.462856001324123 2loss/SRqqt2/assignment_loss‚2.8452325935165086 0loss/FBt3/assignment_loss‚0.6663104715421055 0loss/SRqqt1/detection_loss‚5.401087798751318 1loss/SRqqt4/detection_loss‚1.4654347016587597 .loss/FRt2/detection_loss‚4.101864088740614 2loss/SRqqt1/assignment_loss‚2.9000280705782084 /loss/FRt1/assignment_loss‚5.917593067342585 /loss/FRt2/assignment_loss‚5.346563484933641 2loss/SRqqt3/assignment_loss‚2.8901301499070793 -loss/FBt1/detection_loss‚6.68443460549627 1loss/FBt4/assignment_loss‚0.19938789968586612 /loss/FRt3/detection_loss‚2.8477720436122684 /loss/FBt3/detection_loss‚1.7920469801743002 _runtime‚ 603.885040181 _step‚63 .loss/FRt1/detection_loss‚5.024908365173773 .loss/FRt4/assignment_loss‚4.67454814677145 0loss/FBt2/assignment_loss‚0.4953711274720263 /loss/FRt4/detection_loss‚3.1160296105286656 0loss/SRqqt2/detection_loss‚2.922969181847293 0loss/SRqqt3/detection_loss‚4.768320180218796 /loss/FBt2/detection_loss‚29.964849449694157 epoch‚0 trainer/global_step‚799/C%ðU¿Ï /loss/FRt1/assignment_loss‚5.917593067342585 /loss/FRt2/assignment_loss‚5.346563484933641 2loss/SRqqt3/assignment_loss‚2.8901301499070793 -loss/FBt1/detection_loss‚6.68443460549627 1loss/FBt4/assignment_loss‚0.19938789968586612 /loss/FRt3/detection_loss‚2.8477720436122684 /loss/FBt3/detection_loss‚1.7920469801743002 _runtime‚ 603.885040181 _step‚63 .loss/FRt1/detection_loss‚5.024908365173773 .loss/FRt4/assignment_loss‚4.67454814677145 0loss/FBt2/assignment_loss‚0.4953711274720263 /loss/FRt4/detection_loss‚3.1160296105286656 0loss/SRqqt2/detection_loss‚2.922969181847293 0loss/SRqqt3/detection_loss‚4.768320180218796 /loss/FBt2/detection_loss‚29.964849449694157 epoch‚0 trainer/global_step‚799 $ _timestamp‚1.788395979248281e+09 2loss/SRqqt4/assignment_loss‚2.6868113449641635 0loss/FBt1/assignment_loss‚0.5079044644621068 /loss/FBt4/detection_loss‚2.5807563456948155 &loss/total_loss‚105.26321017409089 /loss/FRt3/assignment_loss‚5.462856001324123 2loss/SRqqt2/assignment_loss‚2.8452325935165086 0loss/FBt3/assignment_loss‚0.6663104715421055 0loss/SRqqt1/detection_loss‚5.401087798751318 1loss/SRqqt4/detection_loss‚1.4654347016587597 .loss/FRt2/detection_loss‚4.101864088740614 2loss/SRqqt1/assignment_loss‚2.9000280705782084?øYÆ4À‚(:’ ÖƒãÔ剮 gpu.0.gpu‚30.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚131.064' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.212799999999998 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0n†–nÁ‚(:ã ÖƒãÔÀâõ°" disk.dm-0.out‚9115.03173828125 disk.rbd1.out‚ 2.486328125' disk.nvme0c0n1.out‚9115.03173828125 disk.drbd1716.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda3.in‚ 0.0078125 network.sent‚978236* disk./.usagePercent‚16.901556480750855 disk.drbd2266.out‚0% disk./.usageGB‚251.84540557861328 disk.drbd2080.out‚0 disk.sda3.out‚ 1195.69140625# proc.memory.availableMB‚2252.25# proc.memory.rssMB‚ 2588.38671875 disk.rbd1.in‚0% disk.nvme0n1.out‚9115.03173828125 disk.dm-1.in‚ 0.0078125 disk.drbd2265.in‚0 disk.dm-0.in‚ 167.07421875 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚86.25335693359375* proc.memory.percent‚15.798258781433105 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 167.07421875 disk.sda.out‚ 1195.69140625 disk.rbd0.out‚0 network.recv‚299691 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.dm-1.out‚ 1195.69140625 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 167.07421875 disk.sda.in‚ 0.0078125 cpu‚25.08512693478023ZLún’‚(:‡ åƒãÔûÕ¬® gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚125.932' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.1864 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ø^œ­sÂ(:è åƒãÔœôü±' disk.nvme0c0n1.out‚9115.23486328125 disk.nvme0n1.in‚ 167.07421875 disk.sda.out‚ 1223.765625 disk.sda2.in‚0 disk.drbd2265.in‚0 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 167.07421875* proc.memory.percent‚15.798258781433105" disk.nvme0c0n1.in‚ 167.07421875 disk.dm-1.out‚ 1223.765625 disk.drbd2266.out‚0 disk.rbd0.in‚0* disk./.usagePercent‚16.901556736758387 disk.sda3.in‚ 0.0078125 disk.rbd1.out‚ 2.486328125' proc.memory.availableMB‚ 2252.484375% disk./.usageGB‚251.84540939331055 disk.sda.in‚ 0.0078125 disk.dm-1.in‚ 0.0078125 disk.drbd2265.out‚0 disk.sda3.out‚ 1223.765625 disk.rbd1.in‚0 disk.sda2.out‚0" disk.dm-0.out‚9115.23486328125 disk.sda1.out‚0 cpu‚25.084237632601138 network.sent‚ 1.028049e+06 disk.sda1.in‚0% disk.nvme0n1.out‚9115.23486328125 disk.drbd1716.out‚0 disk.drbd2080.out‚0$ memory_percent‚86.25192642211914# proc.memory.rssMB‚ 2588.38671875 network.recv‚305118·ñ‚{0Ä+ _wandbruntime‚641  _runtime‚641/;ÆÅÀ ' lr-AdamW/pg1‚0.00015097074853632258 ' lr-AdamW/pg2‚0.00015097074853632258 trainer/global_step‚849 % _timestamp‚1.7883960163901258e+09 _runtime‚ 641.027005072 _step‚642.³4ÊÆÄ ' lr-AdamW/pg1‚0.00015097074853632258 ' lr-AdamW/pg2‚0.00015097074853632258 trainer/global_step‚849 % _timestamp‚1.7883960163901258e+09 _runtime‚ 641.027005072 _step‚64@ì À0Ç+ _wandbruntime‚641  _runtime‚641èýUÀ'ȺO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1183678092288 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10768915 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209254400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚23191605229568 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚23191623232512 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚24375283321856 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1183846325248 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚23242033479822 % _timestamp‚1.7883960163908873e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10960513 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6278656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4681857 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24375283321856 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24375469557760 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16888475 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27657392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚23191623232512 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚24375283321856 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1183846325248 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚23242033479822 % _timestamp‚1.7883960163908873e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚10960513 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6278656 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4681857 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚24375283321856 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚24375469557760 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚16888475 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚27657392 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1183678092288 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚10768915 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209254400 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚23191605229568 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 IDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 trainer/global_step‚849 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚23218874594304 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚24403945824768 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22087508092928 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10781574 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24404132073984 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚23218892597248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1185071230464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10781576 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 _runtime‚ 641.749143115 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27671296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1205613502976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1181840694622 _step‚66 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27689864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4686837 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1185071230464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10781576 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16889722 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚25821954722304 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6286031 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 % _timestamp‚1.7883960171117728e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16908288 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1205822744064 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27689864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10781574 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10973383 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚515 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6286033 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚22087525132288 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1181675580558 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1185239476736 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10972868 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚513 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚23269183673486 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚23218892597248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚23218874594304 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4687350 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚24616108038144 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27671296 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25821721541120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1185239476736 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚10972868 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚513 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚23269183673486 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚23218892597248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚23218874594304 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚4687350 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚24616108038144 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚27671296 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚25821721541120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 trainer/global_step‚849 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚23218874594304 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚24403945824768 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚22087508092928 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚10781574 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚24404132073984 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚23218892597248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1185071230464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚10781576 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 _runtime‚ 641.749143115 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚27671296 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1205613502976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1181840694622 _step‚66 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚27689864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4686837 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1185071230464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚10781576 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚16889722 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚25821954722304 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚6286031 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 % _timestamp‚1.7883960171117728e+09 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚16908288 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1205822744064 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚27689864 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚10781574 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚10973383 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903Bx­Ñï0Í+ _wandbruntime‚641  _runtime‚6413˜–"SÎÍ 0loss/FBt1/assignment_loss‚0.8103455671669162 1loss/FBt4/assignment_loss‚0.22406165164751915 .loss/FRt2/detection_loss‚4.217772227876327 0loss/SRqqt2/detection_loss‚2.754007389396429 trainer/global_step‚849 0loss/FRt4/assignment_loss‚3.6127278094782547 1loss/SRqqt1/assignment_loss‚2.791040778160095 2loss/SRqqt4/assignment_loss‚2.7665989643119904 /loss/FBt3/detection_loss‚7.5758302465674205 2loss/SRqqt2/assignment_loss‚3.0378750824928282 0loss/SRqqt3/detection_loss‚3.234086608602887 1loss/SRqqt4/detection_loss‚0.8472153920869986 .loss/FBt1/detection_loss‚6.427722661977723 1loss/FBt2/assignment_loss‚0.23634400189863905 0loss/SRqqt1/detection_loss‚3.873510618720736 epoch‚0 /loss/FRt2/assignment_loss‚4.783634424209595 /loss/FRt3/assignment_loss‚4.813944089107024 0loss/FBt3/assignment_loss‚0.4762600317480974 /loss/FRt1/detection_loss‚3.8650567531585693 % _timestamp‚1.7883960171134408e+09 /loss/FRt1/assignment_loss‚6.204357060519132 /loss/FRt4/detection_loss‚2.1762467600843487 /loss/FBt2/detection_loss‚12.621623906824324 %loss/total_loss‚84.94842024949936 _runtime‚ 641.750299668 1loss/SRqqt3/assignment_loss‚2.368762072353136 .loss/FRt3/detection_loss‚3.594160408163682 /loss/FBt4/detection_loss‚1.6352357429467024 _step‚67‰í#¶WÏÑ /loss/FBt4/detection_loss‚1.6352357429467024 _step‚67 0loss/FBt1/assignment_loss‚0.8103455671669162 1loss/FBt4/assignment_loss‚0.22406165164751915 .loss/FRt2/detection_loss‚4.217772227876327 0loss/SRqqt2/detection_loss‚2.754007389396429 trainer/global_step‚849 0loss/FRt4/assignment_loss‚3.6127278094782547 1loss/SRqqt1/assignment_loss‚2.791040778160095 2loss/SRqqt4/assignment_loss‚2.7665989643119904 /loss/FBt3/detection_loss‚7.5758302465674205 2loss/SRqqt2/assignment_loss‚3.0378750824928282 0loss/SRqqt3/detection_loss‚3.234086608602887 1loss/SRqqt4/detection_loss‚0.8472153920869986 .loss/FBt1/detection_loss‚6.427722661977723 1loss/FBt2/assignment_loss‚0.23634400189863905 0loss/SRqqt1/detection_loss‚3.873510618720736 epoch‚0 /loss/FRt2/assignment_loss‚4.783634424209595 /loss/FRt3/assignment_loss‚4.813944089107024 0loss/FBt3/assignment_loss‚0.4762600317480974 /loss/FRt1/detection_loss‚3.8650567531585693 % _timestamp‚1.7883960171134408e+09 /loss/FRt1/assignment_loss‚6.204357060519132 /loss/FRt4/detection_loss‚2.1762467600843487 /loss/FBt2/detection_loss‚12.621623906824324 %loss/total_loss‚84.94842024949936 _runtime‚ 641.750299668 1loss/SRqqt3/assignment_loss‚2.368762072353136 .loss/FRt3/detection_loss‚3.594160408163682CH7¿:‘Ђ(:† ôƒãÔƚȰ gpu.0.gpu‚36.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.865' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.373 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÕÚ¼ÈrÑ‚(:ç ôƒãԙò cpu‚25.083800450653598 proc.cpu.threads‚17 disk.sda3.in‚ 0.0078125 disk.sda.out‚ 1275.7421875 disk.dm-1.out‚ 1275.7421875 network.recv‚309462 disk.drbd2265.in‚0 disk.sda.in‚ 0.0078125 disk.drbd1716.in‚0 disk.drbd1716.out‚0 network.sent‚ 1.034155e+06% disk./.usageGB‚251.84541702270508 disk.rbd0.in‚0* proc.memory.percent‚15.798258781433105 disk.drbd2080.in‚0 disk.rbd0.out‚0& disk.nvme0c0n1.out‚9115.6513671875 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0! disk.dm-0.out‚9115.6513671875 disk.rbd1.in‚0 disk.rbd1.out‚ 2.548828125 disk.dm-1.in‚ 0.0078125 disk.drbd2266.in‚0' proc.memory.availableMB‚ 2252.421875 disk.nvme0n1.in‚ 167.07421875* disk./.usagePercent‚16.901557248773457 disk.drbd2265.out‚0 disk.dm-0.in‚ 167.07421875 disk.sda1.in‚0 disk.sda3.out‚ 1275.7421875# memory_percent‚86.2523078918457 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 167.07421875$ disk.nvme0n1.out‚9115.6513671875# proc.memory.rssMB‚ 2588.38671875¶ÎxÒ‚(:… ƒ„ãÔƒÖ—® gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚123.51' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.702 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xêëºrÓ‚(:ç ƒ„ãÔïÈú° network.sent‚ 1.08338e+06) disk./.usagePercent‚16.90155596873579 disk.rbd0.out‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 167.07421875) proc.memory.availableMB‚ 2252.38671875 cpu‚25.082814928811 proc.cpu.threads‚17 disk.sda.out‚ 1314.63671875 disk.dm-1.in‚ 0.0078125 disk.dm-1.out‚ 1314.63671875 disk.drbd2080.in‚0 disk.drbd1716.in‚0 disk.drbd2266.out‚0* proc.memory.percent‚15.798258781433105 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.sda3.in‚ 0.0078125% disk./.usageGB‚251.84539794921875 disk.sda2.out‚0 disk.drbd1716.out‚0# proc.memory.rssMB‚ 2588.38671875 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 1314.63671875 disk.rbd1.in‚0# memory_percent‚86.2525224685669 network.recv‚314751 disk.sda.in‚ 0.0078125 disk.rbd0.in‚0 disk.rbd1.out‚ 2.548828125& disk.nvme0c0n1.out‚9115.9521484375 disk.nvme0n1.in‚ 167.07421875$ disk.nvme0n1.out‚9115.9521484375 disk.drbd2265.in‚0 disk.dm-0.in‚ 167.07421875! disk.dm-0.out‚9115.9521484375tðã7Ô‚(:’ ’„ãÔÏòý­ gpu.0.gpu‚35.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚107.039' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.407799999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ô2Å2vÕ‚(:ë ’„ãÔôꦱ disk.drbd2265.out‚0" disk.dm-0.out‚9116.59716796875 disk.rbd1.in‚0' proc.memory.availableMB‚ 2252.390625! proc.memory.rssMB‚ 2588.390625 network.recv‚318521' disk.nvme0c0n1.out‚9116.59716796875 disk.sda2.in‚0) disk./.usagePercent‚16.90155596873579 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda.in‚ 0.0078125 disk.dm-1.in‚ 0.0078125 disk.drbd2080.out‚0 disk.dm-0.in‚ 167.07421875 disk.sda3.in‚ 0.0078125 disk.rbd1.out‚ 2.564453125 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 1342.44921875* proc.memory.percent‚15.798282623291016 cpu‚25.082443230995757 network.sent‚ 1.088818e+06% disk.nvme0n1.out‚9116.59716796875 disk.sda.out‚ 1342.44921875 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 167.07421875 disk.nvme0n1.in‚ 167.07421875 disk.drbd1716.in‚0 disk.dm-1.out‚ 1342.44921875 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0$ memory_percent‚86.25249862670898% disk./.usageGB‚251.84539794921875 disk.drbd2266.in‚0ÖÅ« 0Ö+ _wandbruntime‚679  _runtime‚679Wµþ>Ä×¾ & lr-AdamW/pg1‚0.0001598618409118422 & lr-AdamW/pg2‚0.0001598618409118422 trainer/global_step‚899 % _timestamp‚1.7883960543963342e+09 _runtime‚ 679.033208836 _step‚68DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11602146 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚24555091469312 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24555091469312 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚25808422881280 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11401867 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11401865 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6647506 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17862637 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17862637 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27308121303552 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29264502 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26032866441216 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚23358741066752 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6647504 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29283044 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚24555073466368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚24608664415030 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1253517647872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1253349414912 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚899 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4955162 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200865792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1249923348278 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚69 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224805888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11401865 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17881177 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚24555073466368 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚25808422881280 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26032842501120 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1253349414912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11401867 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17881177 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1275254862336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚27308121303552 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚29264502 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚26032866441216 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚23358741066752 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚6647504 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚29283044 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚24555073466368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚24608664415030 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1253517647872 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1253349414912 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 trainer/global_step‚899 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚4955162 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200865792 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1249923348278 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚69 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224805888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚11401865 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚17881177 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚24555073466368 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚25808422881280 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚26032842501120 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1253349414912 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚11401867 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚17881177 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1275254862336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚11602146 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚24555091469312 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚24555091469312 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚25808422881280 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚11401867 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚11401865 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚6647506 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚17862637 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚17862637 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 I6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29315516 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6654883 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1254910799360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17882424 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17900990 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1254910799360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1251147398966 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29315516 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚25837271633408 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1254742553088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1276474794496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11414526 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1254742553088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11615148 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚25837271633408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚24582360834048 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4960267 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 205046784 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11414524 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚23384684249088 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27338459230720 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26061779389440 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26061755449344 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1251312513030 _runtime‚ 679.755708179 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29296948 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944•_AÞÇM H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚29315516 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚25837271633408 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1254742553088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1276474794496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚11414526 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1254742553088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚11615148 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚25837271633408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚24582360834048 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚4960267 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 205046784 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚11414524 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚23384684249088 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚27338459230720 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚26061779389440 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚26061755449344 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1251312513030 _runtime‚ 679.755708179 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚29296948 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚29315516 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚6654883 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1254910799360 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚17882424 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚17900990 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1254910799360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1251147398966 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0F™øŸ0ß+ _wandbruntime‚679  _runtime‚679Ú¤dîTàÎ 2loss/SRqqt1/assignment_loss‚2.7865758700804277 $loss/total_loss‚74.1604810858758 /loss/FRt2/assignment_loss‚5.070173358917236 0loss/FRt3/assignment_loss‚4.8435858921571215 2loss/SRqqt4/assignment_loss‚2.4593573573609473 /loss/FRt1/detection_loss‚3.9156819439452626 /loss/FRt3/detection_loss‚3.1748731684955684 1loss/SRqqt2/assignment_loss‚2.250662237405777 1loss/SRqqt3/assignment_loss‚2.515610673427582 1loss/FBt1/assignment_loss‚0.33702595611446834 /loss/FRt2/detection_loss‚2.5131748601794244 trainer/global_step‚899 0loss/FBt2/assignment_loss‚0.8948707594112917 0loss/FBt3/assignment_loss‚0.7370048406032416 .loss/FRt4/detection_loss‚2.458568186987014 /loss/FBt3/detection_loss‚3.1007618095654133 /loss/FRt1/assignment_loss‚6.267793531003206 1loss/FBt4/assignment_loss‚0.24028584082741156 1loss/SRqqt4/detection_loss‚1.2445892582533342 .loss/FBt4/detection_loss‚1.734852482060917 epoch‚0 1loss/SRqqt1/detection_loss‚3.9789094310818296 1loss/SRqqt3/detection_loss‚2.5538762921094893 % _timestamp‚1.7883960551199977e+09 _step‚71 1loss/SRqqt2/detection_loss‚2.6628517946944785 /loss/FRt4/assignment_loss‚3.122779966346801 .loss/FBt1/detection_loss‚5.689697272398255 .loss/FBt2/detection_loss‚9.606918302449314 _runtime‚ 679.756814267¬½—XáÒ /loss/FRt4/assignment_loss‚3.122779966346801 .loss/FBt1/detection_loss‚5.689697272398255 .loss/FBt2/detection_loss‚9.606918302449314 _runtime‚ 679.756814267 2loss/SRqqt1/assignment_loss‚2.7865758700804277 $loss/total_loss‚74.1604810858758 /loss/FRt2/assignment_loss‚5.070173358917236 0loss/FRt3/assignment_loss‚4.8435858921571215 2loss/SRqqt4/assignment_loss‚2.4593573573609473 /loss/FRt1/detection_loss‚3.9156819439452626 /loss/FRt3/detection_loss‚3.1748731684955684 1loss/SRqqt2/assignment_loss‚2.250662237405777 1loss/SRqqt3/assignment_loss‚2.515610673427582 1loss/FBt1/assignment_loss‚0.33702595611446834 /loss/FRt2/detection_loss‚2.5131748601794244 trainer/global_step‚899 0loss/FBt2/assignment_loss‚0.8948707594112917 0loss/FBt3/assignment_loss‚0.7370048406032416 .loss/FRt4/detection_loss‚2.458568186987014 /loss/FBt3/detection_loss‚3.1007618095654133 /loss/FRt1/assignment_loss‚6.267793531003206 1loss/FBt4/assignment_loss‚0.24028584082741156 1loss/SRqqt4/detection_loss‚1.2445892582533342 .loss/FBt4/detection_loss‚1.734852482060917 epoch‚0 1loss/SRqqt1/detection_loss‚3.9789094310818296 1loss/SRqqt3/detection_loss‚2.5538762921094893 % _timestamp‚1.7883960551199977e+09 _step‚71 1loss/SRqqt2/detection_loss‚2.6628517946944785Gû·¸j’â‚(:‡ ¡„ãԧʌ® gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚106.798' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3596 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0O’aAuã‚(:ê ¡„ãÔÇÛ¿±) proc.memory.availableMB‚ 2252.03515625 network.recv‚322713) disk./.usagePercent‚16.90155596873579% disk./.usageGB‚251.84539794921875 disk.drbd2266.out‚0 disk.drbd2265.in‚0" disk.dm-0.out‚9116.86669921875 disk.sda3.in‚ 0.0078125% disk.nvme0n1.out‚9116.86669921875 disk.sda2.out‚0 disk.drbd2080.out‚0 disk.sda3.out‚ 1384.5859375 cpu‚25.082098665628056 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.rbd1.out‚ 2.650390625! proc.memory.rssMB‚ 2588.390625 disk.nvme0n1.in‚ 167.07421875 disk.dm-1.out‚ 1384.5859375 disk.rbd0.in‚0* proc.memory.percent‚15.798282623291016 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 1.094701e+06 disk.dm-1.in‚ 0.0078125 disk.sda1.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.0078125 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 167.07421875' disk.nvme0c0n1.out‚9116.86669921875$ memory_percent‚86.25466823577881 disk.sda.out‚ 1384.5859375 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 167.07421875ÈYÂGä‚(:… °„ãÔÅÌ„® gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚123.51' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.702 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'ÿ?xå‚(:í °„ãÔò©±* proc.memory.percent‚15.798282623291016 disk.drbd2266.in‚0" disk.dm-0.out‚9120.45947265625 disk.rbd1.out‚ 2.712890625 disk.sda2.out‚0 network.sent‚ 1.144451e+06) disk./.usagePercent‚16.90176205480085 disk.dm-1.in‚ 0.0078125 disk.dm-1.out‚ 1432.91015625 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0' disk.nvme0c0n1.out‚9120.45947265625 network.recv‚328185% disk./.usageGB‚251.84846878051758 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚ 0.0078125 disk.sda.out‚ 1432.91015625) proc.memory.availableMB‚ 2252.27734375! proc.memory.rssMB‚ 2588.390625 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 167.07421875 disk.rbd1.in‚0 disk.nvme0n1.in‚ 167.07421875 disk.sda.in‚ 0.0078125 disk.drbd1716.out‚0 cpu‚25.081126528763036 disk.sda1.in‚0 disk.sda1.out‚0% disk.nvme0n1.out‚9120.45947265625 disk.rbd0.in‚0 disk.sda3.out‚ 1432.91015625 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 167.07421875 disk.sda2.in‚0$ memory_percent‚86.25319004058838‚ÖaW0æ+ _wandbruntime‚717  _runtime‚717ÕˆÎkÅç¿ ' lr-AdamW/pg1‚0.00016875293328736177 ' lr-AdamW/pg2‚0.00016875293328736177 trainer/global_step‚949 % _timestamp‚1.7883960928097708e+09 _runtime‚ 717.44662282 _step‚72Fp;Éèà ' lr-AdamW/pg2‚0.00016875293328736177 trainer/global_step‚949 % _timestamp‚1.7883960928097708e+09 _runtime‚ 717.44662282 _step‚72 ' lr-AdamW/pg1‚0.00016875293328736177HR·Uo0é+ _wandbruntime‚717  _runtime‚717"wU ¿'ê¹O _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1345889649152 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7016354 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚24655883144192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚25918541703168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12249072 trainer/global_step‚949 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚25918541703168 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27478513852416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12034817 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12034817 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28824612755968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5232716 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1319395166686 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 $ _timestamp‚1.788396092810473e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12034815 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1319230052830 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1323020737536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚25975295350238 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚27241562440704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚24655900183552 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚27241748676608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1346098903552 _runtime‚ 717.447605535 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚25918559706112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚523 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12034815 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18873879 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27478489912320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚27241748676608 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27241562440704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28824379561472 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30908696 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630¾¿—Ã'ë½O D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚25918541703168 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚27478513852416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12034817 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12034817 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚28824612755968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5232716 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1319395166686 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 $ _timestamp‚1.788396092810473e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚12034815 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1319230052830 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1323020737536 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 220016128 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚25975295350238 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚27241562440704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚24655900183552 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚27241748676608 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1346098903552 _runtime‚ 717.447605535 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚25918559706112 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚523 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚12034815 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚18873879 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚27478489912320 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚27241748676608 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚27241562440704 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚28824379561472 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚30908696 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1345889649152 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7016354 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚24655883144192 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚25918541703168 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚12249072 trainer/global_step‚949 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18893692 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18875126 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7023731 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚25945811067904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1324413875712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12047476 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12261526 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 209241088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚74 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚27270411192832 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_Z}Ppool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚26002445543902 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _runtime‚ 718.160108324 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30941168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 $ _timestamp‚1.788396093522597e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5238321 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1320784331438 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7023733 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1347305928704 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30922600 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12262054 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27507426800640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚24681843365888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚25945829070848 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248|Àä¥Í&îÇM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12047476 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1324413875712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚27507402860544 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚24681826326528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1320619217374 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚28854941970432 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚18893692 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚18875126 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7023731 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚25945811067904 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1324413875712 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12047476 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12261526 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 209241088 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚74 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚27270411192832 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚26002445543902 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _runtime‚ 718.160108324 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚30941168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 220016128 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 $ _timestamp‚1.788396093522597e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5238321 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1320784331438 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7023733 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1347305928704 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚30922600 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12262054 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚27507426800640 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚24681843365888 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚25945829070848 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248J0ï+ _wandbruntime‚718  _runtime‚718de•LðÆ /loss/FRt1/assignment_loss‚5.982800807271685 1loss/SRqqt2/assignment_loss‚2.609860525049012 0loss/FBt1/assignment_loss‚0.4303035715847727 0loss/SRqqt4/detection_loss‚1.339889812268172 /loss/FRt2/assignment_loss‚4.877044622103373 /loss/FRt4/assignment_loss‚2.918194290002187 2loss/SRqqt3/assignment_loss‚2.2661083528529042 .loss/FRt4/detection_loss‚2.315529848138491 .loss/FBt2/detection_loss‚12.08517213538289 _step‚75 /loss/FBt2/assignment_loss‚0.529217436718227 0loss/SRqqt2/detection_loss‚2.434883632536592 -loss/FBt1/detection_loss‚4.99352016672492 0loss/SRqqt1/detection_loss‚3.764712780714035 1loss/SRqqt3/detection_loss‚2.6424572739912118 %loss/total_loss‚73.41764768853537 epoch‚0 .loss/FRt2/detection_loss‚3.099033471610811 .loss/FRt3/detection_loss‚2.463191761502198 /loss/FBt4/detection_loss‚1.4342891913543026 trainer/global_step‚949 _runtime‚ 718.161294383 1loss/FBt4/assignment_loss‚0.34754011665894985 /loss/FRt3/assignment_loss‚4.746507636138371 1loss/SRqqt1/assignment_loss‚2.692932831518578 2loss/SRqqt4/assignment_loss‚2.8011419191772555 0loss/FBt3/assignment_loss‚0.6904419846832752 .loss/FBt3/detection_loss‚2.817965172231238 % _timestamp‚1.7883960935244553e+09 /loss/FRt1/detection_loss‚3.1349083483219147>½ÚøPñÊ %loss/total_loss‚73.41764768853537 epoch‚0 .loss/FRt2/detection_loss‚3.099033471610811 .loss/FRt3/detection_loss‚2.463191761502198 /loss/FBt4/detection_loss‚1.4342891913543026 trainer/global_step‚949 _runtime‚ 718.161294383 1loss/FBt4/assignment_loss‚0.34754011665894985 /loss/FRt3/assignment_loss‚4.746507636138371 1loss/SRqqt1/assignment_loss‚2.692932831518578 2loss/SRqqt4/assignment_loss‚2.8011419191772555 0loss/FBt3/assignment_loss‚0.6904419846832752 .loss/FBt3/detection_loss‚2.817965172231238 % _timestamp‚1.7883960935244553e+09 /loss/FRt1/detection_loss‚3.1349083483219147 /loss/FRt1/assignment_loss‚5.982800807271685 1loss/SRqqt2/assignment_loss‚2.609860525049012 0loss/FBt1/assignment_loss‚0.4303035715847727 0loss/SRqqt4/detection_loss‚1.339889812268172 /loss/FRt2/assignment_loss‚4.877044622103373 /loss/FRt4/assignment_loss‚2.918194290002187 2loss/SRqqt3/assignment_loss‚2.2661083528529042 .loss/FRt4/detection_loss‚2.315529848138491 .loss/FBt2/detection_loss‚12.08517213538289 _step‚75 /loss/FBt2/assignment_loss‚0.529217436718227 0loss/SRqqt2/detection_loss‚2.434883632536592 -loss/FBt1/detection_loss‚4.99352016672492 0loss/SRqqt1/detection_loss‚3.764712780714035 1loss/SRqqt3/detection_loss‚2.6424572739912118KFÜ$ò‚(:’ ¿„ãÔò‡­® gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚123.221' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.644199999999998 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g ïuó‚(:ê ¿„ãÔ¶úØ° network.sent‚ 1.150724e+06 disk.sda2.out‚0$ memory_percent‚86.25514507293701 disk.dm-1.in‚ 0.0078125 disk.rbd0.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 167.07421875 disk.sda.out‚ 1456.2421875" proc.memory.rssMB‚ 2588.3984375 network.recv‚332890$ disk./.usageGB‚251.8487319946289 disk.dm-0.in‚ 167.07421875 disk.sda3.in‚ 0.0078125* proc.memory.percent‚15.798330307006836 cpu‚25.080526051672397 proc.cpu.threads‚17 disk.dm-1.out‚ 1456.2421875 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0" disk.nvme0c0n1.in‚ 167.07421875% disk.nvme0n1.out‚9120.74072265625) disk./.usagePercent‚16.90177971932071 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0" disk.dm-0.out‚9120.74072265625 disk.rbd1.in‚0) proc.memory.availableMB‚ 2251.95703125 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda.in‚ 0.0078125 disk.rbd1.out‚ 2.712890625' disk.nvme0c0n1.out‚9120.74072265625 disk.sda1.in‚0 disk.sda3.out‚ 1456.2421875 disk.sda2.in‚0Ô/] ›ô‚(: ΄ãÔæ® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚114.6' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.919999999999998 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0c˸fõ‚(:Û ΄ãÔÈÆ¿± disk.rbd1.in‚0 disk.rbd1.out‚ 2.7451171875 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.in‚0% disk.nvme0c0n1.out‚9121.173828125 disk.nvme0n1.in‚ 167.07421875 disk.drbd2265.in‚0% disk./.usageGB‚251.84871292114258 disk.sda3.in‚ 0.0078125 disk.sda2.out‚0 disk.drbd2080.in‚0$ memory_percent‚86.25376224517822 network.recv‚337762 disk.dm-0.in‚ 167.07421875 disk.dm-0.out‚9121.173828125 disk.sda1.out‚0 disk.sda.in‚ 0.0078125 disk.dm-1.out‚1491.625 disk.drbd2080.out‚0" proc.memory.rssMB‚ 2588.3984375* disk./.usagePercent‚16.901778439283042 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 167.07421875 disk.sda.out‚1491.625 disk.dm-1.in‚ 0.0078125 disk.drbd2266.out‚0# disk.nvme0n1.out‚9121.173828125 disk.drbd1716.out‚0 disk.drbd2266.in‚0) proc.memory.availableMB‚ 2252.18359375 disk.drbd1716.in‚0 disk.drbd2265.out‚0* proc.memory.percent‚15.798330307006836 cpu‚25.079868180667553 network.sent‚ 1.199694e+06 disk.sda3.out‚1491.625 disk.sda2.in‚0HaÍQö‚(:’ Ý„ãÔÝË¡® gpu.0.gpu‚29.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚127.287' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.457400000000003 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}â3§r÷‚(:ç Ý„ãÔûÙ̱ disk.sda1.out‚0 disk.drbd1716.in‚0 network.sent‚ 1.205662e+06 network.recv‚342262% disk./.usageGB‚251.84871292114258 disk.sda.in‚ 0.0078125 disk.dm-1.in‚ 0.0078125 disk.dm-1.out‚ 1524.8046875 disk.sda3.out‚ 1524.8046875 disk.nvme0n1.in‚ 167.07421875 cpu‚25.078960597733225" disk.nvme0c0n1.in‚ 167.07421875 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.dm-0.in‚ 167.07421875 disk.sda3.in‚ 0.0078125 disk.rbd1.out‚ 2.7451171875" proc.memory.rssMB‚ 2588.3984375$ memory_percent‚86.25380992889404 proc.cpu.threads‚17# disk.nvme0n1.out‚9121.376953125 disk.sda.out‚ 1524.8046875 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda2.in‚0* proc.memory.percent‚15.798330307006836% disk.nvme0c0n1.out‚9121.376953125 disk.drbd2265.in‚0 disk.rbd1.in‚0) proc.memory.availableMB‚ 2252.17578125* disk./.usagePercent‚16.901778439283042 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.out‚9121.376953125¢\2—0ø+ _wandbruntime‚755  _runtime‚755Õ2®øÆùÀ ' lr-AdamW/pg1‚0.00017764402566288135 ' lr-AdamW/pg2‚0.00017764402566288135 trainer/global_step‚999 % _timestamp‚1.7883961307052824e+09 _runtime‚ 755.342165544 _step‚76zA¥rÊúÄ % _timestamp‚1.7883961307052824e+09 _runtime‚ 755.342165544 _step‚76 ' lr-AdamW/pg1‚0.00017764402566288135 ' lr-AdamW/pg2‚0.00017764402566288135 trainer/global_step‚999L A¹,0û+ _wandbruntime‚755  _runtime‚755¢×F•!ü¹O [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27282009939968 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 222112256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 229000192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1416958222848 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32515806 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28674888236032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1392692060160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 $ _timestamp‚1.788396130706019e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28674702000128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _step‚77 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚30340890486272 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12891400 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚27282027942912 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27282009939968 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27282027942912 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1392860293120 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7385206 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28674888236032 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12667767 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19866581 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 205060096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚550 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28924137323520 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚25953059300352 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.large_p¯ñ¶S*ool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚27341744132230 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1388866985094 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1388701871238 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 _runtime‚ 755.343131568 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12667767 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1392692060160 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1416753162752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚28924161263616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚25953042260992 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904ºß)Ã'ý½O E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚32515806 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚28674888236032 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1392692060160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 $ _timestamp‚1.788396130706019e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚28674702000128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _step‚77 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚30340890486272 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚12891400 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚27282027942912 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚27282009939968 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚27282027942912 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1392860293120 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7385206 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚28674888236032 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚12667767 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚19866581 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 205060096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚550 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚28924137323520 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚25953059300352 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚27341744132230 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1388866985094 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1388701871238 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 _runtime‚ 755.343131568 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚12667767 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1392692060160 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1416753162752 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚28924161263616 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚25953042260992 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚27282009939968 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 222112256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 229000192 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1416958222848 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248M®¶ 0þ+ _wandbruntime‚756  _runtime‚756¼K(Ê&ÿÄM D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32566820 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27369076479110 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _runtime‚ 756.063658473 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 239472640 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1390256149846 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1394253444608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5512209 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1394085198336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 trainer/global_step‚999 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32548252 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19867828 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚25979002482688 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚27309279304704 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28703550752256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27309297307648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚30371215669248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12680426 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚27309279304704 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7392581 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚27309297307648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚28953050271744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 222112256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7392583 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19886394 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1394253444608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12904242 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚548 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28703550752256 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12680426 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12680424 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12904792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1418380930048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚78 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27369258632534 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚25978985443328 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19886394 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5511661 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1418165397504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚28703364503040 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28953074211840 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32566820 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7883961314262373e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5512209 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1394085198336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 trainer/global_step‚999 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚32548252 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚19867828 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚25979002482688 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚27309279304704 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚28703550752256 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚27309297307648 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚30371215669248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚12680426 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚27309279304704 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7392581 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚27309297307648 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚28953050271744 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 222112256 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7392583 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚19886394 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1394253444608 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚12904242 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚548 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚28703550752256 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚12680426 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚12680424 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚12904792 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1418380930048 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚78 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚27369258632534 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚25978985443328 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚19886394 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5511661 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1418165397504 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚28703364503040 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚28953074211840 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚32566820 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7883961314262373e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚32566820 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚27369076479110 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _runtime‚ 756.063658473N†¡0+ _wandbruntime‚756  _runtime‚756‰ âÊê‚Ê 2loss/SRqqt1/assignment_loss‚2.3738961562514307 /loss/FRt1/assignment_loss‚5.686682488237109 /loss/FRt1/detection_loss‚2.9905031621456146 .loss/FRt2/detection_loss‚3.645450024141206 1loss/SRqqt2/detection_loss‚2.5931028935771723 .loss/FBt1/detection_loss‚5.111633121967316 /loss/FBt2/detection_loss‚7.6307651201883955 % _timestamp‚1.7883961314279976e+09 /loss/FRt3/assignment_loss‚4.585295912623406 0loss/FBt2/assignment_loss‚0.5502254365216382 0loss/FBt4/assignment_loss‚0.3594684547173198 _step‚79 .loss/FRt2/assignment_loss‚4.94151085946295 0loss/FRt4/assignment_loss‚2.9572084414756903 2loss/SRqqt4/assignment_loss‚2.4570085681974887 0loss/SRqqt4/detection_loss‚1.471571449635303 /loss/FBt4/detection_loss‚1.4480646781532525 0loss/FBt3/assignment_loss‚0.3425329864607193 0loss/SRqqt1/detection_loss‚3.077727011591196 1loss/FBt1/assignment_loss‚0.44577392575213354 /loss/FRt4/detection_loss‚2.325813046703904€šo«f4 0loss/SRqqt3/detection_loss‚2.538281041522359 epoch‚0 trainer/global_step‚999 _runtime‚ 756.064789277 .loss/FBt3/detection_loss‚8.099985858798027 %loss/total_loss‚74.21711159939862 2loss/SRqqt2/assignment_loss‚2.7356469264397254 0loss/SRqqt3/assignment_loss‚2.63545207090156 .loss/FRt3/detection_loss‚3.213511963933706æÓù£TƒÎ .loss/FRt2/assignment_loss‚4.94151085946295 0loss/FRt4/assignment_loss‚2.9572084414756903 2loss/SRqqt4/assignment_loss‚2.4570085681974887 0loss/SRqqt4/detection_loss‚1.471571449635303 /loss/FBt4/detection_loss‚1.4480646781532525 0loss/FBt3/assignment_loss‚0.3425329864607193 0loss/SRqqt1/detection_loss‚3.077727011591196 1loss/FBt1/assignment_loss‚0.44577392575213354 /loss/FRt4/detection_loss‚2.3258130467039044 0loss/SRqqt3/detection_loss‚2.538281041522359 epoch‚0 trainer/global_step‚999 _runtime‚ 756.064789277 .loss/FBt3/detection_loss‚8.099985858798027 %loss/total_loss‚74.21711159939862 2loss/SRqqt2/assignment_loss‚2.7356469264397254 0loss/SRqqt3/assignment_loss‚2.63545207090156 .loss/FRt3/detection_loss‚3.213511963933706 2loss/SRqqt1/assignment_loss‚2.3738961562514307 /loss/FRt1/assignment_loss‚5.686682488237109 /loss/FRt1/detection_loss‚2.9905031621456146 .loss/FRt2/detection_loss‚3.645450024141206 1loss/SRqqt2/detection_loss‚2.5931028935771723 .loss/FBt1/detection_loss‚5.111633121967316 /loss/FBt2/detection_loss‚7.6307651201883955 % _timestamp‚1.7883961314279976e+09 /loss/FRt3/assignment_loss‚4.585295912623406 0loss/FBt2/assignment_loss‚0.5502254365216382 0loss/FBt4/assignment_loss‚0.3594684547173198 _step‚79OZ©X„‚(:’ ì„ãÔ䚈® gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.088' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.217599999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0C 8\}…‚(:ò ì„ãÔ“Íö° network.sent‚ 1.211684e+06 disk.dm-1.out‚ 1548.55078125 disk.rbd0.out‚0 disk.rbd1.out‚ 2.80810546875* disk./.usagePercent‚16.901778439283042 disk.sda.in‚ 0.0078125 disk.drbd1716.in‚0 disk.dm-0.in‚ 167.07421875 disk.sda3.out‚ 1548.55078125 disk.drbd2266.out‚0' disk.nvme0c0n1.out‚9121.93896484375# proc.memory.rssMB‚ 2588.40234375 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 167.07421875$ memory_percent‚86.25438213348389* proc.memory.percent‚15.798354148864746 proc.cpu.threads‚17% disk.nvme0n1.out‚9121.93896484375 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 2252.08203125% disk./.usageGB‚251.84871292114258 disk.rbd0.in‚0" disk.dm-0.out‚9121.93896484375 disk.sda3.in‚ 0.0078125 cpu‚25.078456182049404 disk.sda.out‚ 1548.55078125 disk.dm-1.in‚ 0.0078125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 network.recv‚346448 disk.nvme0n1.in‚ 167.07421875 disk.drbd2080.out‚0/lV‘†‚(:† û„ãÔÊÙ±® gpu.0.gpu‚20.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚127.435' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.487 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šÙ{‡‚(:ð û„ãÔ䀶± disk.rbd1.out‚ 2.80810546875 disk.drbd2080.out‚0 cpu‚25.077893020974198 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.dm-1.out‚ 1581.29296875* disk./.usagePercent‚16.901778439283042 disk.rbd0.in‚0" disk.dm-0.out‚9122.22802734375' disk.nvme0c0n1.out‚9122.22802734375 disk.drbd2266.out‚0 disk.drbd2265.out‚0% disk./.usageGB‚251.84871292114258$ memory_percent‚86.25593185424805' proc.memory.availableMB‚ 2251.828125# proc.memory.rssMB‚ 2588.40234375* proc.memory.percent‚15.798354148864746 disk.rbd1.in‚0 disk.nvme0n1.in‚ 167.07421875 disk.sda.out‚ 1581.29296875 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.0078125 disk.drbd2265.in‚0 network.recv‚352509 disk.rbd0.out‚0 disk.dm-0.in‚ 167.07421875 disk.sda1.out‚0 disk.sda3.out‚ 1581.29296875" disk.nvme0c0n1.in‚ 167.07421875 disk.sda2.in‚0 disk.drbd2266.in‚0 network.sent‚ 1.261672e+06 disk.sda1.in‚0 disk.sda3.in‚ 0.0078125% disk.nvme0n1.out‚9122.22802734375 disk.sda2.out‚0 disk.sda.in‚ 0.0078125 disk.drbd2080.in‚0›D?0ˆ+ _wandbruntime‚793  _runtime‚793ì­ ÙƉÀ ' lr-AdamW/pg1‚0.00018653511803840093 ' lr-AdamW/pg2‚0.00018653511803840093 trainer/global_step‚1049 $ _timestamp‚1.788396168866997e+09 _runtime‚ 793.503901913 _step‚80äLÁÊŠÄ $ _timestamp‚1.788396168866997e+09 _runtime‚ 793.503901913 _step‚80 ' lr-AdamW/pg1‚0.00018653511803840093 ' lr-AdamW/pg2‚0.00018653511803840093 trainer/global_step‚1049P4Y´¿0‹+ _wandbruntime‚793  _runtime‚793aWÈtÁ'Œ»O A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30108027795456 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 _step‚81 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30107841559552 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7754056 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5781430 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚28645496179712 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1462363382784 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20859283 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚ 793.504947466 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30107841559552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31857659820032 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚28708375067438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7883961688678334e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7754054 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚28708557220654 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34160000 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30108027795456 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20859283 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1462531615744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1458338803502 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218514432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚28645496179712 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13534996 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27250201377792 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20840743 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30369784734720 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚27250218417152 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13300715 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚1049 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13300717 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13535486 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚31857441305600 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1487851145216 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20840743 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1462363382784 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30369808674816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416ø8kÅ'¿O UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30108027795456 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚20859283 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1462531615744 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1458338803502 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 218514432 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚28645496179712 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚13534996 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚27250201377792 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚20840743 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚30369784734720 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚27250218417152 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13300715 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚1049 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13300717 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚13535486 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚31857441305600 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1487851145216 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚20840743 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1462363382784 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚30369808674816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚30108027795456 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 _step‚81 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚30107841559552 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚7754056 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚5781430 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚28645496179712 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1462363382784 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚20859283 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚ 793.504947466 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚30107841559552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚31857659820032 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚28708375067438 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7883961688678334e+09 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚7754054 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚28708557220654 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚34160000 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0Qlg|0Ž+ _wandbruntime‚794  _runtime‚794ÂDÚ ÄM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚28672747541504 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚34173904 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚30398697682944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13313376 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1463756520960 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1463924767232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31887772124672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 205046784 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1459562854190 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13313374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20879096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 222112256 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚28735707414318 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13313376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28672747541504 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30136504062464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1489279488512 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20879096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7883961695855787e+09 _step‚82 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚27276161599488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34192472 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _runtime‚ 794.22295681 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20860530 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚530 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5786257 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚30136690311680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27276144560128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34173904 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7761431 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5786787 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20860530 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1463756520960 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34192472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1459727968254 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚28672765544448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13547688 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13313374 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7761433 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 trainer/global_step‚1049 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31888001111552 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚20860530 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1463756520960 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚34192472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1459727968254 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚28672765544448 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚13547688 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚13313374 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7761433 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 trainer/global_step‚1049 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚31888001111552 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚31887772124672 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 205046784 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1459562854190 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13313374 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚20879096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 222112256 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚28735707414318 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13313376 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚28672747541504 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚30136504062464 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1489279488512 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚20879096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7883961695855787e+09 _step‚82 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚27276161599488 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚34192472 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _runtime‚ 794.22295681 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚20860530 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚530 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚5786257 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚30136690311680 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚27276144560128 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚34173904 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚7761431 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚5786787 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096R©¯e0‘+ _wandbruntime‚794  _runtime‚794¤ÿ“T’Î _runtime‚ 794.224240373 /loss/FRt3/assignment_loss‚4.112389166246761 1loss/SRqqt1/assignment_loss‚2.279100540009412 1loss/FBt1/assignment_loss‚0.44169712508612846 epoch‚0 /loss/FRt1/assignment_loss‚5.313105225563049 .loss/FRt1/detection_loss‚2.550514507293701 1loss/SRqqt3/detection_loss‚2.6074535234106913 /loss/FBt1/detection_loss‚3.8626161089965274 /loss/FBt4/detection_loss‚1.1098994489165448 2loss/SRqqt4/assignment_loss‚2.3684516838618688 0loss/SRqqt4/detection_loss‚1.703253186508248 /loss/FBt3/detection_loss‚6.5750819683074955 % _timestamp‚1.7883961695874379e+09 0loss/FBt4/assignment_loss‚0.3838787926193925 /loss/FRt4/detection_loss‚2.3952010765119836 /loss/FRt2/assignment_loss‚4.349109800238359 1loss/SRqqt3/assignment_loss‚2.225070991449886 2loss/SRqqt2/assignment_loss‚2.2558632559246488 0loss/FBt3/assignment_loss‚0.4264069416410166 _step‚83 /loss/FRt4/assignment_loss‚3.473029599145607 .loss/FRt2/detection_loss‚3.425999085369863 1loss/SRqqt1/detection_loss‚3.4146309531096257 $loss/total_loss‚69.8708705459822 trainer/global_step‚1049 1loss/FBt2/assignment_loss‚0.47128000474021975 /loss/FRt3/detection_loss‚2.8745717480778694 1loss/SRqqt2/detection_loss‚2.7648865481217704 .loss/FBt2/detection_loss‚8.487379264831542-SäVX“Ò % _timestamp‚1.7883961695874379e+09 0loss/FBt4/assignment_loss‚0.3838787926193925 /loss/FRt4/detection_loss‚2.3952010765119836 /loss/FRt2/assignment_loss‚4.349109800238359 1loss/SRqqt3/assignment_loss‚2.225070991449886 2loss/SRqqt2/assignment_loss‚2.2558632559246488 0loss/FBt3/assignment_loss‚0.4264069416410166 _step‚83 /loss/FRt4/assignment_loss‚3.473029599145607 .loss/FRt2/detection_loss‚3.425999085369863 1loss/SRqqt1/detection_loss‚3.4146309531096257 $loss/total_loss‚69.8708705459822 trainer/global_step‚1049 1loss/FBt2/assignment_loss‚0.47128000474021975 /loss/FRt3/detection_loss‚2.8745717480778694 1loss/SRqqt2/detection_loss‚2.7648865481217704 .loss/FBt2/detection_loss‚8.487379264831542 _runtime‚ 794.224240373 /loss/FRt3/assignment_loss‚4.112389166246761 1loss/SRqqt1/assignment_loss‚2.279100540009412 1loss/FBt1/assignment_loss‚0.44169712508612846 epoch‚0 /loss/FRt1/assignment_loss‚5.313105225563049 .loss/FRt1/detection_loss‚2.550514507293701 1loss/SRqqt3/detection_loss‚2.6074535234106913 /loss/FBt1/detection_loss‚3.8626161089965274 /loss/FBt4/detection_loss‚1.1098994489165448 2loss/SRqqt4/assignment_loss‚2.3684516838618688 0loss/SRqqt4/detection_loss‚1.703253186508248 /loss/FBt3/detection_loss‚6.5750819683074955S‚àûÒ’”‚(:‡ Š…ãÔüò¨® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7332 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Üpm¨x•‚(:í Š…ãÔÛžÕ° disk.sda.out‚ 1627.70703125 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 167.07421875 disk.drbd2080.out‚0* proc.memory.percent‚15.798354148864746 disk.rbd1.in‚0 disk.drbd1716.in‚0 proc.cpu.threads‚17 network.sent‚ 1.267033e+06 disk.sda2.in‚0 disk.dm-1.out‚ 1627.70703125 network.recv‚355945% disk./.usageGB‚252.65848922729492 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.dm-0.in‚ 167.07421875 disk.nvme0n1.in‚ 167.07421875 disk.dm-1.in‚ 0.0078125 disk.drbd2080.in‚0 cpu‚25.077126024063343( disk./.usagePercent‚16.9561232066365 disk.sda3.out‚ 1627.70703125 disk.drbd2265.out‚0" disk.dm-0.out‚9330.32958984375$ memory_percent‚86.25633716583252 disk.sda3.in‚ 0.0078125 disk.rbd1.out‚ 2.87109375 disk.sda2.out‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 2251.76171875# proc.memory.rssMB‚ 2588.40234375 disk.sda1.out‚0' disk.nvme0c0n1.out‚9330.32958984375% disk.nvme0n1.out‚9330.32958984375 disk.sda.in‚ 0.0078125õÀ’%‘–‚(:† ™…ãÔØí«® gpu.0.gpu‚20.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.485' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.297 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0}üW€z—‚(:ï ™…ãÔï€ú± disk.drbd2080.in‚0 disk.dm-0.in‚ 182.03515625" disk.nvme0c0n1.in‚ 182.03515625* proc.memory.percent‚15.798354148864746 cpu‚25.076858063356287 network.sent‚ 1.316581e+06 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda.in‚ 0.0078125 disk.sda2.out‚0 proc.cpu.threads‚17 network.recv‚360998& disk.nvme0n1.out‚12973.55615234375 disk.sda2.in‚0# memory_percent‚86.2572193145752 disk.sda3.in‚ 0.0078125( disk.nvme0c0n1.out‚12973.55615234375 disk.dm-1.out‚ 1668.01953125 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 1668.01953125 disk.drbd1716.out‚0( proc.memory.availableMB‚ 2251.6171875 disk.sda1.out‚0 disk.sda.out‚ 1668.01953125 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.0078125* disk./.usagePercent‚17.289523866537905 disk.rbd0.in‚0# disk.dm-0.out‚12973.55615234375$ disk./.usageGB‚257.6263999938965 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 2.95703125 disk.nvme0n1.in‚ 182.03515625# proc.memory.rssMB‚ 2588.40234375òDÇrœ˜‚(:‘ ¨…ãÔøõƒ® gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚125.643' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.128600000000002 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³´ÄËx™‚(:í ¨…ãÔ¼÷‹± disk.nvme0n1.in‚ 182.03515625# proc.memory.rssMB‚ 2588.40234375 proc.cpu.threads‚17$ disk./.usageGB‚261.2790298461914( disk.nvme0c0n1.out‚15931.14794921875 disk.rbd0.out‚0 disk.sda3.out‚ 1701.03515625( proc.memory.availableMB‚ 2251.3671875 network.recv‚365321 disk.sda2.in‚0 disk.dm-1.out‚ 1701.03515625 disk.drbd2266.in‚0 disk.drbd2080.in‚0# disk.dm-0.out‚15931.14794921875$ memory_percent‚86.25874519348145* proc.memory.percent‚15.798354148864746 disk.sda.in‚ 0.0078125 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.0078125 disk.drbd2265.out‚0 disk.dm-0.in‚ 182.03515625 disk.sda3.in‚ 0.0078125 disk.rbd1.in‚0 network.sent‚ 1.322575e+06 disk.sda.out‚ 1701.03515625 disk.drbd2266.out‚0 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 3.01953125 cpu‚25.076436493574956' disk./.usagePercent‚17.534654920686 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 182.03515625& disk.nvme0n1.out‚15931.14794921875 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0>á‡0š+ _wandbruntime‚831  _runtime‚831%¹%Æ›À trainer/global_step‚1099 $ _timestamp‚1.788396206693299e+09 _runtime‚ 831.330314004 _step‚84 ' lr-AdamW/pg1‚0.00019542621041392054 ' lr-AdamW/pg2‚0.00019542621041392054\c‚¶ÊœÄ ' lr-AdamW/pg2‚0.00019542621041392054 trainer/global_step‚1099 $ _timestamp‚1.788396206693299e+09 _runtime‚ 831.330314004 _step‚84 ' lr-AdamW/pg1‚0.00019542621041392054TD*ÿB0+ _wandbruntime‚831  _runtime‚831[ÝöÍÀ'žºO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35785652 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚31541167354880 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6056375 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1558746236416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1532034705408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 222112256 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚31540981118976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31540981118976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚28547377533952 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14179810 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209254400 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚30008964416512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13933665 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13933667 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13933665 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚30075188155862 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚31541167354880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1532202938368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31815432145920 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚28547360494592 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8122904 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚30008964416512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14179279 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233194496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31815456086016 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1532034705408 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35767110 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 trainer/global_step‚1099 $ _timestamp‚1.788396206694092e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21833445 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1527810621910 _step‚85 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚30008946413568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚30008946413568 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚33374202322432 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚33373969127936 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8122906 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6056904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30075006002646 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1532202938368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1527645508054 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 831.331230975 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13933667 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21851985 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21851985 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21833445 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1558536982016 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0‰çH‚TŸ¾O 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 209254400 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚30008964416512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚13933665 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757237709 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚13933667 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚13933665 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚30075188155862 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚31541167354880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1532202938368 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_b þŒ«p!ytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚31815432145920 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚28547360494592 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8122904 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚30008964416512 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14179279 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 233194496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚31815456086016 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1532034705408 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚35767110 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 trainer/global_step‚1099 $ _timestamp‚1.788396206694092e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚21833445 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1527810621910 _step‚85 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚30008946413568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚30008946413568 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚33374202322432 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚33373969127936 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8122906 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6056904 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚30075006002646 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1532202938368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1527645508054 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚ 831.331230975 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚13933667 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚21851985 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚21851985 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚21833445 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1558536982016 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168232960 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚35785652 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚31541167354880 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6056375 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1558746236416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1532034705408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 222112256 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚31540981118976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚31540981118976 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚28547377533952 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14179810 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416Urqñ0 + _wandbruntime‚832  _runtime‚832KŽÊdË&¡ÅM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1533596089856 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35799556 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚28573320716288 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13946324 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14192002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚30036233781248 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31844345094144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1533596089856 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1533427843584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6061721 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21853232 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21871798 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1529034672598 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 trainer/global_step‚1099 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚31569829871104 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚33404529572864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1560160538624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚ 832.045709594 _step‚86 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚30036233781248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚30036215778304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7883962074081664e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚503 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚33404304780288 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30102520502950 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35818124 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13946326 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚30036215778304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1559959686144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚31569829871104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35799556 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21853232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13946326 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚505 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35818124 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30102338349526 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 222112256 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚28573303676928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709)½rÏ&¢ÉM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14192002 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚30036233781248 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚31844345094144 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1533596089856 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1533427843584 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6061721 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚21853232 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚21871798 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1529034672598 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 trainer/global_step‚1099 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚31569829871104 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚33404529572864 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1560160538624 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚ 832.045709594 _step‚86 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚30036233781248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚30036215778304 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7883962074081664e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚503 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚33404304780288 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚30102520502950 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚35818124 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚18568 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚13946326 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚30036215778304 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1559959686144 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚31569829871104 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚35799556 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚21853232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚13946326 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚505 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚35818124 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚30102338349526 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 168246272 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 222112256 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚28573303676928 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1533596089856 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚35799556 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚28573320716288 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚13946324VÝlƒJ0£+ _wandbruntime‚832  _runtime‚832¹sü·Q¤Ë _runtime‚ 832.046889471 0loss/FRt3/assignment_loss‚4.2113679258190855 2loss/SRqqt1/assignment_loss‚2.0115044362015193 1loss/FBt1/assignment_loss‚0.43993665454137143 .loss/FBt2/detection_loss‚8.758693801032173 epoch‚0 0loss/FBt2/assignment_loss‚0.3799741700180069 0loss/FBt4/assignment_loss‚0.3223831159816958 1loss/SRqqt1/detection_loss‚2.3853417158126833 0loss/FRt4/assignment_loss‚2.7914794837986983 1loss/SRqqt4/assignment_loss‚1.939544677734375 /loss/FRt3/detection_loss‚2.5202776911307354 /loss/FBt4/detection_loss‚1.9141152296528197 /loss/FRt1/assignment_loss‚5.180063601463072 0loss/FBt3/assignment_loss‚0.2613005089845371 /loss/FRt1/detection_loss‚3.2642208308942857 -loss/FRt2/detection_loss‚2.72756290435791 1loss/SRqqt2/detection_loss‚2.5787283726360486 1loss/SRqqt3/detection_loss‚2.2656463936288307 .loss/FBt1/detection_loss‚5.454976749089029 _step‚87 /loss/FRt4/detection_loss‚2.3329362932730606 trainer/global_step‚1099 $ _timestamp‚1.788396207410124e+09 .loss/FRt2/assignment_loss‚4.92668301363786 1loss/SRqqt2/assignment_loss‚2.567198757244193 0loss/SRqqt3/assignment_loss‚1.94405157451934 1loss/SRqqt4/detection_loss‚1.7859028959790102 .loss/FBt3/detection_loss‚7.143423872334616 %loss/total_loss‚70.107314669764956žêÍU¥Ï 1loss/SRqqt2/assignment_loss‚2.567198757244193 0loss/SRqqt3/assignment_loss‚1.94405157451934 1loss/SRqqt4/detection_loss‚1.7859028959790102 .loss/FBt3/detection_loss‚7.143423872334616 %loss/total_loss‚70.10731466976495 _runtime‚ 832.046889471 0loss/FRt3/assignment_loss‚4.2113679258190855 2loss/SRqqt1/assignment_loss‚2.0115044362015193 1loss/FBt1/assignment_loss‚0.43993665454137143 .loss/FBt2/detection_loss‚8.758693801032173 epoch‚0 0loss/FBt2/assignment_loss‚0.3799741700180069 0loss/FBt4/assignment_loss‚0.3223831159816958 1loss/SRqqt1/detection_loss‚2.3853417158126833 0loss/FRt4/assignment_loss‚2.7914794837986983 1loss/SRqqt4/assignment_loss‚1.939544677734375 /loss/FRt3/detection_loss‚2.5202776911307354 /loss/FBt4/detection_loss‚1.9141152296528197 /loss/FRt1/assignment_loss‚5.180063601463072 0loss/FBt3/assignment_loss‚0.2613005089845371 /loss/FRt1/detection_loss‚3.2642208308942857 -loss/FRt2/detection_loss‚2.72756290435791 1loss/SRqqt2/detection_loss‚2.5787283726360486 1loss/SRqqt3/detection_loss‚2.2656463936288307 .loss/FBt1/detection_loss‚5.454976749089029 _step‚87 /loss/FRt4/detection_loss‚2.3329362932730606 trainer/global_step‚1099 $ _timestamp‚1.788396207410124e+09 .loss/FRt2/assignment_loss‚4.92668301363786W·Žãi’¦‚(:‡ ·…ãÔÔ‘„® gpu.0.gpu‚20.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚125.932' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.1864 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wœÏg§‚(:ï ·…ãÔˈ˜° disk.drbd2265.out‚0 disk.rbd1.out‚ 3.01953125 disk.sda2.out‚0 disk.dm-1.out‚ 1727.48046875 disk.drbd2266.out‚0 disk.dm-0.in‚ 314.16015625 disk.sda1.out‚0 disk.drbd1716.out‚0( proc.memory.availableMB‚ 2251.5546875 proc.cpu.threads‚17% disk./.usageGB‚260.56737518310547& disk.nvme0n1.out‚17895.55029296875 disk.drbd1716.in‚0 network.recv‚370095 disk.sda3.in‚ 0.0078125" disk.nvme0c0n1.in‚ 314.16015625( disk.nvme0c0n1.out‚17895.55029296875 disk.nvme0n1.in‚ 314.16015625 disk.sda.out‚ 1727.48046875 disk.dm-1.in‚ 0.0078125 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.drbd2266.in‚0$ memory_percent‚86.25760078430176# proc.memory.rssMB‚ 2588.40234375* proc.memory.percent‚15.798354148864746 cpu‚25.07574357565631) disk./.usagePercent‚17.48689517912827 disk.sda.in‚ 0.0078125 network.sent‚ 1.328739e+06 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda3.out‚kNC3[ 1727.48046875 disk.rbd1.in‚0 disk.rbd0.in‚0# disk.dm-0.out‚17895.55029296875q~ý𑨂(:† Æ…ãÔÞè® gpu.0.gpu‚28.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.625' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.325 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¡ñZïr©‚(:ç Æ…ãÔæ€Ç° disk.sda3.out‚ 1769.171875 disk.rbd1.in‚0 disk.sda.out‚ 1769.171875 disk.sda2.out‚0 disk.drbd2080.in‚0$ memory_percent‚86.25783920288086# proc.memory.rssMB‚ 2588.40234375 disk.drbd1716.in‚0 disk.drbd2265.in‚0) disk./.usagePercent‚17.48766934591181( disk.nvme0c0n1.out‚17907.00439453125 disk.sda.in‚ 0.01171875 disk.sda1.in‚0 disk.sda1.out‚0 network.sent‚ 1.378012e+06$ disk./.usageGB‚260.5789108276367 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.dm-0.in‚ 334.1484375 disk.sda3.in‚ 0.01171875 disk.dm-1.out‚ 1769.171875' proc.memory.availableMB‚ 2251.515625 proc.cpu.threads‚17 disk.rbd1.out‚ 3.16796875 disk.nvme0n1.in‚ 334.1484375& disk.nvme0n1.out‚17907.00439453125 disk.rbd0.out‚0* proc.memory.percent‚15.798354148864746! disk.nvme0c0n1.in‚ 334.1484375 disk.sda2.in‚0 disk.drbd2080.out‚0# disk.dm-0.out‚17907.00439453125 network.recv‚375122 disk.dm-1.in‚ 0.01171875 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 cpu‚25.07499891157927z3ª‚(:’ Õ…ãÔ󻺮 gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚101.136' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.227199999999996 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01Jôšs«‚(:è Õ…ãԃĮ±( disk.nvme0c0n1.out‚18126.63330078125 disk.sda2.out‚0 disk.drbd1716.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd1.out‚ 3.16796875 disk.sda.out‚ 1813.06640625 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 1813.06640625 disk.dm-1.out‚ 1813.06640625 network.sent‚ 1.38401e+06 network.recv‚379511 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 334.1484375# memory_percent‚86.2579345703125* disk./.usagePercent‚17.483460326041364$ disk./.usageGB‚260.5161933898926 disk.dm-0.in‚ 334.1484375 disk.nvme0n1.in‚ 334.1484375& disk.nvme0n1.out‚18126.63330078125 disk.dm-1.in‚ 0.01171875 cpu‚25.074812800555776 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.sda3.in‚ 0.01171875 disk.drbd1716.in‚0" proc.memory.availableMB‚2251.5 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda2.in‚0* proc.memory.percent‚15.798354148864746# disk.dm-0.out‚18126.63330078125 disk.sda.in‚ 0.01171875# proc.memory.rssMB‚ 2588.40234375DŒÂ0¬+ _wandbruntime‚870  _runtime‚870Y3ÂÇ­Á ' lr-AdamW/pg1‚0.00020431730278944012 ' lr-AdamW/pg2‚0.00020431730278944012 trainer/global_step‚1149 % _timestamp‚1.7883962460450544e+09 _runtime‚ 870.681878595 _step‚88â¿Ë®Å ' lr-AdamW/pg1‚0.00020431730278944012 ' lr-AdamW/pg2‚0.00020431730278944012 trainer/global_step‚1149 % _timestamp‚1.7883962460450544e+09 _runtime‚ 870.681878595 _step‚88X"Yy0¯+ _wandbruntime‚870  _runtime‚870ˆmþÁ'°»O _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚29844536650752 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1601874260992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1629603143680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14824564 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1629395986432 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚32974120678400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚31372432653312 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22826147 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33261103497216 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚32974120678400 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 231097344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚89 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14825118 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚31372432653312 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37411304 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6332810 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 222112256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8491754 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22844687 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22826147 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1601874260992 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1597117326462 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29844519611392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚ 870.682985274 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37392762 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚32974306914304 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7883962460457962e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚32974306914304šH Å'±¿O C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚ 870.682985274 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚37392762 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚32974306914304 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7883962460457962e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚32974306914304 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚29844536650752 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1601874260992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1629603143680 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚14824564 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1629395986432 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚32974120678400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚31372432653312 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚22826147 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚33261103497216 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚32974120678400 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 186235904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 231097344 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 409945088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _step‚89 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚14825118 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚31372432653312 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168232960 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚37411304 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6332810 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 222112256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 186235904 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8491754 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚22903 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚22844687 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚22826147 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1601874260992 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1597117326462 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚29844519611392 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554Yã70²+ _wandbruntime‚871  _runtime‚871B*n!³ÅM E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚31399702018048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231084032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1598671605070 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 trainer/global_step‚1149 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29870462793728 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚31399702018048 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1603267412480 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8499131 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1598506491006 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37443776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22845934 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14837939 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1603267412480 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37425208 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14579276 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22845934 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8499133 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6338281 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33289992505344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14837412 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37443776 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1631024751104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚31399684015104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14579274 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 207143936 _step‚90 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22864500 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requ±+<ested_bytes.small_pool.current‚ 165114064 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22864500 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6338806 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚31468969284734 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33002969430528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14579274 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0¨¯$Ï&´ÉM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚37425208 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚14579276 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚22845934 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8499133 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6338281 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚33289992505344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚14837412 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182153424 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚37443776 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1631024751104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚31399684015104 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚14579274 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 207143936 _step‚90 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚22864500 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚22864500 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚6338806 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚31468969284734 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚33002969430528 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚14579274 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚31399702018048 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231084032 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1598671605070 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 trainer/global_step‚1149 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚29870462793728 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚31399702018048 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1603267412480 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚18566 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8499131 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1598506491006 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚37443776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚22845934 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚14837939 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1603267412480Z™×0µ+ _wandbruntime‚871  _runtime‚871Ä:èV¶Ð 1loss/FBt3/assignment_loss‚0.21792276622727513 1loss/FBt4/assignment_loss‚0.23697875916246858 0loss/SRqqt3/detection_loss‚2.603641217781438 1loss/SRqqt4/detection_loss‚1.7156438616092329 epoch‚0 0loss/FRt1/assignment_loss‚4.5752806551754475 /loss/FRt4/detection_loss‚2.3296051144599916 .loss/FBt3/detection_loss‚8.438963398337364 %loss/total_loss‚65.71299934340783 /loss/FBt4/detection_loss‚1.1028061738189006 0loss/FRt4/assignment_loss‚3.0923328854820946 1loss/SRqqt3/assignment_loss‚2.107874374422762 2loss/SRqqt4/assignment_loss‚2.2100940474442075 0loss/FBt2/assignment_loss‚0.5046841380317346 /loss/FRt1/detection_loss‚2.6606065668165684 % _timestamp‚1.7883962467992897e+09 _runtime‚ 871.436135599 /loss/FRt3/detection_loss‚2.4809301924705505 1loss/SRqqt1/detection_loss‚2.7071113854646685 0loss/SRqqt2/detection_loss‚2.336727283748926 .loss/FBt1/detection_loss‚3.992276809785677 /loss/FRt2/assignment_loss‚4.347719422795556 /loss/FRt3/assignment_loss‚4.183470585346222 2loss/SRqqt2/assignment_loss‚2.5335087600876305 0loss/FBt1/assignment_loss‚0.4743461864949811 trainer/global_step‚1149 2loss/SRqqt1/assignment_loss‚2.1883116573095323 /loss/FRt2/detection_loss‚2.8722214834256605 -loss/FBt2/detection_loss‚5.79994161770894 _step‚91¢¦7Z·Ô 0loss/FRt1/assignment_loss‚4.5752806551754475 /loss/FRt4/detection_loss‚2.3296051144599916 .loss/FBt3/detection_loss‚8.438963398337364 %loss/total_loss‚65.71299934340783 /loss/FBt4/detection_loss‚1.1028061738189006 0loss/FRt4/assignment_loss‚3.0923328854820946 1loss/SRqqt3/assignment_loss‚2.107874374422762 2loss/SRqqt4/assignment_loss‚2.2100940474442075 0loss/FBt2/assignment_loss‚0.5046841380317346 /loss/FRt1/detection_loss‚2.6606065668165684 % _timestamp‚1.7883962467992897e+09 _runtime‚ 871.436135599 /loss/FRt3/detection_loss‚2.4809301924705505 1loss/SRqqt1/detection_loss‚2.7071113854646685 0loss/SRqqt2/detection_loss‚2.336727283748926 .loss/FBt1/detection_loss‚3.992276809785677 /loss/FRt2/assignment_loss‚4.347719422795556 /loss/FRt3/assignment_loss‚4.183470585346222 2loss/SRqqt2/assignment_loss‚2.5335087600876305 0loss/FBt1/assignment_loss‚0.4743461864949811 trainer/global_step‚1149 2loss/SRqqt1/assignment_loss‚2.1883116573095323 /loss/FRt2/detection_loss‚2.8722214834256605 -loss/FBt2/detection_loss‚5.79994161770894 _step‚91 1loss/FBt3/assignment_loss‚0.21792276622727513 1loss/FBt4/assignment_loss‚0.23697875916246858 0loss/SRqqt3/detection_loss‚2.603641217781438 1loss/SRqqt4/detection_loss‚1.7156438616092329 epoch‚0[¬Žæ[’¸‚(:‡ ä…ãÔýÜ©® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚31.0 gpu.0.powerWatts‚101.254' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.2508 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ʘH|¹‚(:ñ ä…ãÔ°‚‡± cpu‚25.07442108315924 disk.sda3.out‚ 1844.09765625 disk.drbd1716.in‚0 disk.drbd1716.out‚0( proc.memory.availableMB‚ 2251.4453125 proc.cpu.threads‚17* disk./.usagePercent‚17.483055834137264# disk.dm-0.out‚18153.10595703125 disk.nvme0n1.in‚ 334.1484375& disk.nvme0n1.out‚18153.10595703125 disk.sda2.out‚0 disk.drbd2266.out‚0# proc.memory.rssMB‚ 2588.40234375 network.sent‚ 1.390001e+06 disk.rbd1.out‚ 3.2001953125 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.sda3.in‚ 0.01171875! disk.nvme0c0n1.in‚ 334.1484375 disk.dm-1.in‚ 0.01171875$ disk./.usageGB‚260.5101661682129 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.dm-1.out‚ 1844.09765625 disk.drbd2265.out‚0 disk.dm-0.in‚ 334.1484375 disk.sda1.out‚0 disk.sda.in‚ 0.01171875 disk.drbd2265.in‚0* proc.memory.percent‚15.798354148864746 network.recv‚383942 disk.rbd1.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚18153.10595703125 disk.sda.out‚ 1844.09765625$ memory_percent‚86.25826835632324cž”I’º‚(:‡ ó…ãÔÁÈ® gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚105.154' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.0308 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿÒ*=z»‚(:ï ó…ãÔ…ˆ´±! disk.nvme0c0n1.in‚ 334.1484375 disk.sda.out‚ 1889.42578125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-1.out‚ 1889.42578125$ memory_percent‚86.25669479370117# proc.memory.rssMB‚ 2588.40234375' disk.nvme0c0n1.out‚18154.3974609375% disk.nvme0n1.out‚18154.3974609375" disk.dm-0.out‚18154.3974609375 network.sent‚ 1.440008e+06 network.recv‚390213 disk.drbd2266.in‚0 disk.sda3.out‚ 1889.42578125 disk.sda1.in‚0 disk.sda3.in‚ 0.01171875 disk.sda.in‚ 0.01171875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 334.1484375 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd1.out‚ 3.2001953125 disk.nvme0n1.in‚ 334.1484375 disk.dm-1.in‚ 0.01171875' proc.memory.availableMB‚ 2251.703125* proc.memory.percent‚15.798354148864746* disk./.usagePercent‚17.483056858167405% disk./.usageGB‚260.51018142700195 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 cpu‚25.074004592024714 disk.drbd2265.in‚0 disk.sda1.out‚0b>ȯ0¼+ _wandbruntime‚910  _runtime‚910‚I#½¼ trainer/global_step‚1199 $ _timestamp‚1.788396285732289e+09 _runtime‚ 910.3692454 _step‚92 & lr-AdamW/pg1‚0.0002132083951649597 & lr-AdamW/pg2‚0.0002132083951649597¿HaqƾÀ $ _timestamp‚1.788396285732289e+09 _runtime‚ 910.3692454 _step‚92 & lr-AdamW/pg1‚0.0002132083951649597 & lr-AdamW/pg2‚0.0002132083951649597 trainer/global_step‚1199\Í#C0¿+ _wandbruntime‚910  _runtime‚910žÏNÀ'ÀºO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23818849 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15199565 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23837389 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚512 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34706726968320 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34407260237824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1671545583616 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23818849 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚36406992159744 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8860606 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚34407446473728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200865792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6609789 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23837389 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15470393 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚1199 _runtime‚ 910.370206194 _step‚93 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚36407216965632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15199565 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6610301 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 222112256 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15470907 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8860604 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1671377350656 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚31141678728192ƒ{¨êÁ¾O \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚18540 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚32735900890112 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚34407260237824 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚15199567 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚32735882887168 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1671377350656 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚34407446473728 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚18540 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚18542 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4911254016 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚22903 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚15470907 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚8860604 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406782421 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1671377350656 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚31141678728192 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚23818849 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚15199565 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚23837389 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4911254016 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚512 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚34706726968320 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182153216 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚22630 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚34407260237824 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1671545583616 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚23818849 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚36406992159744 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 409945088 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚8860606 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚22630 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165113856 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚34407446473728 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200865792 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚6609789 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚18542 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚23837389 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚15470393 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 trainer/global_step‚1199 _runtime‚ 910.370206194 _step‚93 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚36407216965632 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚15199565 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚6610301 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 222112256 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39069428 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15483122 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚94 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39050860 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1672938735104 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23857202 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚34435922740736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚32763152251904 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1672770488832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚32835782373366 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23838636 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6615141 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39069428 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 222112256 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8867981 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15212226 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1701680965120 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚32763152251904 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8867983 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚34436108989952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1701888109056 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚31167638949888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231084032 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚32763170254848 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1672938735104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚31167621910528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1667978309414 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15483636 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0ëžÇÎ&ÄÈM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚18566 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 409945088 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚8867981 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚15212226 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406782421 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚22630 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚22903 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1701680965120 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚32763152251904 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚22903 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚8867983 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 186249216 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4911254016 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 168246272 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚34436108989952 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1701888109056 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚31167638949888 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 231084032 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚32763170254848 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1672938735104 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 409945088 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚31167621910528 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1667978309414 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚15483636 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚34436108989952 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚32763170254848 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚23838636 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚36437320881664 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757237709 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚39069428 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚15483122 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165114064 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚94 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚39050860 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 186249216 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1672938735104 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚23857202 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚34435922740736 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚32763152251904 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1672770488832 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚32835782373366 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚23838636 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚6615141 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚39069428 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4911254016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 222112256 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 O ÃttÜbo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak"trainer/global_step2:JPi@eÝb` =DeviceStatsMonitor.on_validation_batch_start/num_device_alloc"trainer/global_step2:JPõE\tÞbo LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak"trainer/global_step2:JPõaKtßbo LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak"trainer/global_step2:JP©L¼gàbb ?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections"trainer/global_step2:JP§(Ðnábi FDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed"trainer/global_step2:JP×'Ôlâbg DDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed"trainer/global_step2:JP»””™lãbg DDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed"trainer/global_step2:JPš42òwäbr ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current"trainer/global_step2:JP‰ë„ómåbh EDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed"trainer/global_step2:JP'Ñ7Önæbi FDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current"trainer/global_step2:JPfà ÿwçbr ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current"trainer/global_step2:JP†6>kèbf CDeviceStatsMonitor.on_validation_batch_start/allocation.all.current"trainer/global_step2:JPå¸7zrébm JDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak"trainer/global_step2:JPÇݹŒyêbt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPæÞE$eëb` =DeviceStatsMonitor.on_validation_batch_start/active.all.freed"trainer/global_step2:JPô"tìbo LDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current"trainer/global_step2:JP`»‰xíbs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated"trainer/global_step2:JP¾ìH4sîbn KDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak"trainer/global_step2:JPA¦ø‡|ïbw TDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current"trainer/global_step2:JP9%â/dðb_ tøbo LDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed"trainer/global_step2:JPMÝþqùbl IDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current"trainer/global_step2:JPú8Ûkúbf CDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed"trainer/global_step2:JP×¼Ivûbq NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated"trainer/global_step2:JPÁùuübp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed"trainer/global_step2:JPCÍÄ,týbo LDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated"trainer/global_step2:JP, ¿žtþbo LDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed"trainer/global_step2:JP7Ž)fÿba >DeviceStatsMonitor.on_validation_batch_start/segment.all.freed"trainer/global_step2:JPÓˆ,Üm€bh EDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed"trainer/global_step2:JPhÞežqbl IDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak"trainer/global_step2:JP·fgr‚bm JDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current"trainer/global_step2:JP'=5oƒbj GDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current"trainer/global_step2:JP蟞·q„bl IDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated"trainer/global_step2:JP€Ö…v…bq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current"trainer/global_step2:JP™ƒµm†bh EDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak"trainer/global_step2:JP¹L)Ìo‡bj GDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed"trainer/global_step2:JP~Üàpˆbk HDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated"trainer/global_step2:JP-¿x‰bs PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated"trainer/global_step2:JP©lhŠbc @DeviceStatsMonitor.on_validation_batch_start/segment.all.current"trainer/global_step2:JPó}d s‹bn KDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak"trainer/global_step2:JP26ËÿrŒbm JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed"trainer/global_step2:JP¦ŽW­obj GDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak"trainer/global_step2:JP ÜfõoŽbj GDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current"trainer/global_step2:JP$ƒE®mbh EDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed"trainer/global_step2:JPÔ¹dŽsbn KDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak"trainer/global_step2:JP\vo4q‘bl IDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated"trainer/global_step2:JPÄ/Öd’b_ ,*j›be BDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated"trainer/global_step2:JPšÔXÄcœb^ ;DeviceStatsMonitor.on_validation_batch_start/max_split_size"trainer/global_step2:JPUâºwbr ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current"trainer/global_step2:JP¼ÂBvžbq NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current"trainer/global_step2:JP¾hï0qŸbl IDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated"trainer/global_step2:JP™4Gu bp MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed"trainer/global_step2:JP‰!>m¡bh EDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed"trainer/global_step2:JP ×ìmy¢bt QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak"trainer/global_step2:JPíd8Nn£bi FDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current"trainer/global_step2:JP-Çž-t¤bo LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak"trainer/global_step2:JPUôún¥bi FDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current"trainer/global_step2:JPá“É p¦bk HDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current"trainer/global_step2:JP«xNdl§bg DDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak"trainer/global_step2:JP¼œqx¨bs PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated"trainer/global_step2:JP¬9¿Íy©bt QDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated"trainer/global_step2:JPˆ]Лuªbp MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current"trainer/global_step2:JP}V²W~«by VDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated"trainer/global_step2:JP%>¯>k¬bf CDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak"trainer/global_step2:JP?˜p­bk HDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current"trainer/global_step2:JP´Î¡—k®bf CDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak"trainer/global_step2:JP¤zÂm¯bh EDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current"trainer/global_step2:JPÒÏ1ëi°bd ADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed"trainer/global_step2:JP2GÌ›t±bo LDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed"trainer/global_step2:JPµ¶V"t²bo LDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated"trainer/global_step2:JPà” q³bl IDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated"trainer/global_step2:JPÝÈ_2z´bu RDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed"trainer/global_step2:JPK@gµbb ?DeviceStatsMonitor.on_validation_batch_start/active.all.current"trainer/global_step2:JP¤¼žo¶bj GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated"trainer/global_step2:JPÞM)‰m·bh EDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated"trainer/global_step2:JPëôû”~¸by VDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JPܶáÞ|¹bw TDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current"trainer/global_step2:JP9 `lsºbn KDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated"trainer/global_step2:JPE{VQu»bp MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed"trainer/global_step2:JPRåTh¼bc @DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak"trainer/global_step2:JPÏ×EÛp½bk HDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed"trainer/global_step2:JP\©nv¾bq NDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current"trainer/global_step2:JPªøãp¿bk HDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated"trainer/global_step2:JP(é,~lÀbg DDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak"trainer/global_step2:JPf¯ÂlÁbg DDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak"trainer/global_step2:JP¯quÆyÂbt QDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated"trainer/global_step2:JPNQŒÐwÃbr ODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated"trainer/global_step2:JPÆ=h…sÄbn KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed"trainer/global_step2:JPßû¿zÅbu RDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed"trainer/global_step2:JP#‚tÆbo LDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed"trainer/global_step2:JPmô²iÇbd ADeviceStatsMonitor.on_validation_batch_start/active.all.allocated"trainer/global_step2:JPíÛ ÉqÈbl IDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak"trainer/global_step2:JPÿ8lÉbg DDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak"trainer/global_step2:JPúЭfÊba >DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries"trainer/global_step2:JP•Á ß0Ë+ _wandbruntime‚970  _runtime‚970®’P5(*Ì¢T hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 1731801919488 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 1697570706950 trainer/global_step‚1208 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚22903 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚15851043 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚282 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚217 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚199 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 1702599595520 bODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚37688257022976 ^KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚37688129302016 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚35956431163392 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚40248687 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 409945088 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1003 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 _step‚96 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 4911254016 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚32278286681088 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚217 _runtime‚ 970.846305146 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 126905856 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚24382451 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚33975998349714 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚24397642 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 1702472689664 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚33932988641280 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 409945088 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚416 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 4390226944 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 YFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚33975857388038 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚24382451 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 123922316 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15191 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15193 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 262255616 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚40233494 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 \PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚9252876 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚982 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 103780864 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 4541568000 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15191 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚199 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 5041553408 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 4911254016 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚15851045 \PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚6910977 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚26 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚33932970638336 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚22630 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚284 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚217 % _timestamp‚1.7883963462087948e+09 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 _step‚96 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 4911254016 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚32278286681088 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚217 _runtime‚ 970.846305146 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 126905856 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚24382451 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚33975998349714 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚24397642 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 1702472689664 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚33932988641280 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 409945088 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚416 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 4390226944 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 YFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚33975857388038 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚24382451 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 123922316 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15191 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15193 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 262255616 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚40233494 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 \PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_Fâ^¥"pool.allocated‚9252876 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚982 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 103780864 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 4541568000 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15191 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚199 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 5041553408 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 4911254016 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚15851045 \PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚6910977 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚26 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚33932970638336 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚22630 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚284 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚217 % _timestamp‚1.7883963462087948e+09 FDeviceStatsMonitor.on_validation_batch_end/segment.all.current"trainer/global_step2:JP€æ3rrãbm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed"trainer/global_step2:JPuzÊýxäbs PDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed"trainer/global_step2:JP$§ýbbåb] :DeviceStatsMonitor.on_validation_batch_end/active.all.peak"trainer/global_step2:JP­ËRgjæbe BDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed"trainer/global_step2:JPÀªÞÈrçbm JDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed"trainer/global_step2:JPüúórèbm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak"trainer/global_step2:JPêA }jébe BDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak"trainer/global_step2:JPé/Æsêbn KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed"trainer/global_step2:JPm¦H|jëbe BDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak"trainer/global_step2:JPO`‚pìbk HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated"trainer/global_step2:JPâ/Äfsíbn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed"trainer/global_step2:JPí@ìêsîbn KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed"trainer/global_step2:JP\«:xmïbh EDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current"trainer/global_step2:JP-JEjðbe BDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed"trainer/global_step2:JP€øírñbm JDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated"trainer/global_step2:JPª×I\oòbj GDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current"trainer/global_step2:JP$?"äjóbe BDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak"trainer/global_step2:JPs¢lôbg DDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed"trainer/global_step2:JPÀªènõbi FDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated"trainer/global_step2:JPîÛ3möbh EDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated"trainer/global_step2:JPÊ^½æf÷ba >DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak"trainer/global_step2:JP®ë«.wøbr ODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated"trainer/global_step2:JPÏ',™tùbo LDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current"trainer/global_step2:JPmÍÊkjúbe BDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed"trainer/global_step2:JP ¶wÀmûbh EDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current"trainer/global_step2:JP¤9Îàkübf CDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current"trainer/global_step2:JP0 –ïlýbg DDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current"trainer/global_step2:JP£~Øu€bp MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current"trainer/global_step2:JP7åá4obj GDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak"trainer/global_step2:JPw\·;|‚bw TDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated"trainer/global_step2:JP7Ë€Flƒbg DDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current"trainer/global_step2:JPòü¡t„bo LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated"trainer/global_step2:JP"ÁZÜu…bp MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current"trainer/global_step2:JPìÕ×#p†bk HDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current"trainer/global_step2:JP«¿Ñ²g‡bb ?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated"trainer/global_step2:JP@‹Iópˆbk HDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak"trainer/global_step2:JPçô¨‘w‰br ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak"trainer/global_step2:JPXŽ gzŠbu RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current"trainer/global_step2:JP]g>r‹bm JDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak"trainer/global_step2:JPDÍ·eŒb` =DeviceStatsMonitor.on_validation_batch_end/active.all.current"trainer/global_step2:JP·¸ùëtbo LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current"trainer/global_step2:JPÅ°änŽbi FDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated"trainer/global_step2:JP$ËÓ@pbk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed"trainer/global_step2:JPòí«rbm JDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak"trainer/global_step2:JP Øv‘bq NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated"trainer/global_step2:JPX8mo’bj GDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated"trainer/global_step2:JP.q1ãc“b^ ;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc"trainer/global_step2:JPm0b”b] :DeviceStatsMonitor.on_validation_batch_end/num_device_free"trainer/global_step2:JP”Ï p•bk HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed"trainer/global_step2:JP™®Þs–bn KDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current"trainer/global_step2:JPP»ñ!h—bc @DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated"trainer/global_step2:JPâÊ ed˜b_ DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 VJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚9259374 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 1732047274496 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚22903 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚15861755 VJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚6915787 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 5041553408 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚33951835824128 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1003 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚33994247509714 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚32296188234752 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 1697918313286 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚217 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚24389847 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚24405038 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚217 WDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚35654657649664 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚15861757 XEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚35654802558464 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚22903 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚32296205274112 _runtime‚ 971.871816801 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚283 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 5041553408^ÎÒå Ë°R VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 4541568000 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 1732151055360 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15191 ^LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚ 1702948731392 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 L?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚40266795 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 262255616 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 103780864 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 406782421 trainer/global_step‚1208 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15193 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚24405038 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚33951835824128 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚199 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚416 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 1702821825536 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 127720960 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚416 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 VJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚9259374 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 1732047274496 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚22903 OBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚15861755 VJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚6915787 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 5041553408 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚33951835824128 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1003 [HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚33994247509714 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚32296188234752 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 1697918313286 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚217 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚24389847 WJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚24405038 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚217 WDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚35654657649664 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚15861757 XEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚35654802558464 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚22903 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚32296205274112 _runtime‚ 971.871816801 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚283 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 5041553408 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 144908800 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚285 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360adŸ“Ì‚(:… ¾†ãÔ ¿®® gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.625' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð)ªówÍ‚(:ì ¾†ãÔűÁ± disk.sda3.in‚0.015625 disk.rbd1.out‚ 3.44384765625* disk./.usagePercent‚17.483044825813295 disk.dm-1.out‚ 2138.65234375 disk.drbd2265.in‚0 disk.drbd2080.out‚0! disk.nvme0c0n1.in‚ 334.1484375( disk.nvme0c0n1.out‚18162.50048828125 disk.nvme0n1.in‚ 334.1484375 disk.sda2.out‚0# memory_percent‚86.8741512298584% disk./.usageGB‚260.51000213623047 disk.sda.in‚0.015625 disk.sda2.in‚0 disk.dm-1.in‚0.015625 disk.rbd0.in‚0 disk.rbd0.out‚0# disk.dm-0.out‚18162.50048828125 disk.sda1.out‚0 disk.sda1.in‚0& disk.nvme0n1.out‚18162.50048828125( proc.memory.availableMB‚ 2150.5390625 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda3.out‚ 2138.65234375 disk.rbd1.in‚0 network.sent‚ 1.512875e+06 disk.drbd1716.in‚0 disk.drbd2080.in‚0" proc.memory.rssMB‚ 2688.4453125 cpu‚25.349462961536176 network.recv‚411778 disk.sda.out‚ 2138.65234375 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.dm-0.in‚ 334.1484375* proc.memory.percent‚16.408967971801758 proc.cpu.threads‚17>ØèJÎbE "Purity/4FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP0HeÐJÏbE "Purity/4FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPˆaÞYHÐbC Purity/4FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP”?‚nNÑbI &Purity/1FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÕ=é9NÒbI &Purity/2FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPEëNÓbI &Purity/4FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPÜJï ;Ôb6 jet/accuracy_2_of_6"trainer/global_step2:JPGÌÎJÕbE "Purity/1FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPC¯†ÅHÖbC Purity/*FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPBü$QH×bC Purity/*FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP„Å|NØbI &Purity/0FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPÅÞi¶HÙbC Purity/2FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPuíÙ>JÚbE "Purity/3FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÞ.é4HÛbC Purity/1FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPþ¹¥0JÜbE "Purity/4FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPH"‰ÚHÝbC Purity/0FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPu¿ñNÞbI &Purity/0FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP3¥¸íJßbE "Purity/3FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÔèHàbC Purity/*FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPIKÑÐJábE "Purity/1FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP‚È"‡NâbI &Purity/1FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP3|6INãbI &Purity/2FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPõè‘NäbI &Purity/3FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPÕ÷ÐëNåbI &Purity/4FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPù Á.JæbE "Purity/0FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPlÔ^>JçbE "Purity/0FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPʯZŸHèbC Purity/2FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP\­…bNébI &Purity/2FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPjYP¬HêbC Purity/2FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JP#9³/NëbI &Purity/2FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPUÄHìbC Purity/2FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPéÆ NíbI &Purity/2FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP¸sÓÐJîbE "Purity/2FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPujýJïbE "Purity/*FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPÙ­ÉbNðbI &Purity/0FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JP™‹RŠJñbE "Purity/1FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP>6sNòbI &Purity/3FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPÎÿ·NóbI &Purity/4FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPéh?­JôbE "Purity/*FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPù0HõbC Purity/3FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPï‰ÈŒHöbC Purity/4FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPsH÷bC Purity/*FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JP¢3X#NøbI &Purity/*FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPsⱆHùbC Purity/*FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JP Y}¯HúbC Purity/1FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP½FÉ‹HûbC Purity/2FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPô@4dNübI &Purity/2FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JP¡{(JýbE "Purity/*FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPzÅ 5;þb6 jet/accuracy_4_of_6"trainer/global_step2:JP «ÅîHÿbC Purity/0FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPåÙ_÷N€bI &Purity/0FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP8Ãþ+HbC Purity/3FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPxKRJ‚bE "Purity/0FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP²`ÁçHƒbC Purity/1FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP@ïEH„bC Purity/3FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPKàÀH…bC Purity/3FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JPÈ£ûJJ†bE "Purity/3FRt1SRqqt0FBt/event_purity"trainer/global_step2:JPIÿˆH‡bC Purity/*FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPFáX°NˆbI &Purity/*FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP-…7€N‰bI &Purity/1FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JP‰dûõJŠbE "Purity/2FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP1ŸûSH‹bC Purity/2FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPE!x¬HŒbC Purity/*FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JP·ÎùiJbE "Purity/1FRt1SRqqt4FBt/event_purity"trainer/global_step2:JP°P3âJŽbE "Purity/*FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP‰rL HbC Purity/0FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JP5دNbI &Purity/2FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPÙQ£ ;‘b6 jet/accuracy_5_of_6"trainer/global_step2:JPó¯Q+H’bC Purity/3FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPçR1íJ“bE "Purity/0FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¾P ÑJ”bE "Purity/0FRt1SRqqt3FBt/event_purity"trainer/global_step2:JP%"ôÊJ•bE "Purity/4FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP” ˆH–bC Purity/*FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP:“K>N—bI &Purity/0FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPy;Û!H˜bC Purity/2FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JP ¨ê§H™bC Purity/2FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP@¸´ÇHšbC Purity/*FRt*SRqqt4FBt/FRt_purity"trainer/global_step2:JP²'£>J›bE "Purity/3FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP1Bó®HœbC Purity/*FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPoE5HbC Purity/*FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPíW ¡NžbI &Purity/*FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JP–SòLNŸbI &Purity/0FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JPhî8N bI &Purity/1FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPæËvZH¡bC Purity/3FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP‰ ¨£N¢bI &Purity/*FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JPË„¼›N£bI &Purity/0FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP£ÐN¤bI &Purity/0FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPahp¡J¥bE "Purity/2FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP†ØQN¦bI &Purity/2FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP“µ-J§bE "Purity/2FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPßzܲJ¨bE "Purity/*FRt3SRqqt1FBt/event_purity"trainer/global_step2:JPñå|8H©bC Purity/*FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP^°HªbC Purity/1FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP«æ N«bI &Purity/4FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP²…ËJ¬bE "Purity/3FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP Ïw¨H­bC Purity/3FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP^´taN®bI &Purity/3FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPÊï"tJ¯bE "Purity/*FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPÄD~N°bI &Purity/*FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPô-áN±bI &Purity/1FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP^°ð•H²bC Purity/1FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPwÃ0WJ³bE "Purity/3FRt1SRqqt*FBt/event_purity"trainer/global_step2:JP§02²H´bC Purity/3FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPáÝ;9HµbC Purity/*FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JP½7ãóN¶bI &Purity/0FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPñ2H·bC Purity/3FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPxf³H¸bC Purity/1FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JPÚôòÿH¹bC Purity/3FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPt¢ ìHºbC Purity/0FRt2SRqqt3FBt/FBt_purity"trainer/global_step2:JP¾Ã<¿N»bI &Purity/0FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP„ÇÇ@H¼bC Purity/1FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP4ØsdH½bC Purity/2FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP•}ÛÃH¾bC Purity/2FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP!ªìqN¿bI &Purity/3FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP&Ñ* HÀbC Purity/1FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JPBY´±HÁbC Purity/3FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPý‰ŠNÂbI &Purity/*FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP[úÚNÃbI &Purity/*FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP?1dØNÄbI &Purity/0FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPå4´JÅbE "Purity/1FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPînlQJÆbE "Purity/2FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPáqцNÇbI &Purity/2FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP péö@Èb; particle/accuracy_2_of_7"trainer/global_step2:JP¼LuNÉbI &Purity/*FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPº'~JÊbE "Purity/0FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP´)@ÀHËbC Purity/3FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JP‹Ò+JÌbE "Purity/3FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPM©‚çHÍbC Purity/1FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JPOtNÎbI &Purity/3FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPFäVHÏbC Purity/*FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPZºJÐbE "Purity/0FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP µ@“HÑbC Purity/1FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPÙÿp­HÒbC Purity/1FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JP¢aŠÂNÓbI &Purity/3FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPnšDœNÔbI &Purity/2FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JP˜ÛýJÕbE "Purity/*FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¡ú:HÖbC Purity/3FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPêPÛJ×bE "Purity/*FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP¸28JØbE "Purity/1FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP`õg³JÙbE "Purity/1FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP§ ÄçHÚbC Purity/2FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPØ}òßJÛbE "Purity/2FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP4¨èHÜbC Purity/0FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JP‚˜JÝbE "Purity/0FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÚ¨Ù–NÞbI &Purity/0FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPæ!ŽNßbI &Purity/3FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPn9•HàbC Purity/*FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP)çJábE "Purity/*FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPj[NâbI &Purity/*FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPÆpF/NãbI &Purity/0FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÄöƒQNäbI &Purity/0FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP¤Q«øJåbE "Purity/3FRt2SRqqt1FBt/event_purity"trainer/global_step2:JP©Ô;æb6 jet/accuracy_5_of_7"trainer/global_step2:JPšˆGHçbC Purity/3FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPØÒ—JèbE "Purity/*FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP⥧ðJébE "Purity/*FRt2SRqqt*FBt/event_purity"trainer/global_step2:JP,‘°JêbE "Purity/0FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¢&›HëbC Purity/2FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPwõf8ìb3 particle/f_score"trainer/global_step2:JPòUvâJíbE "Purity/0FRt4SRqqt1FBt/event_purity"trainer/global_step2:JP\hk @îb; particle/accuracy_1_of_7"trainer/global_step2:JPµ‚iÚJïbE "Purity/1FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPÁ©`¯JðbE "Purity/*FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP yHñbC Purity/1FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP» ·`JòbE "Purity/1FRt2SRqqt1FBt/event_purity"trainer/global_step2:JPõiS~NóbI &Purity/*FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JP¬Æ&ÛJôbE "Purity/*FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP¶;ÞJõbE "Purity/2FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPp•X¿HöbC Purity/0FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP…5~¹H÷bC Purity/2FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP ‘¤6NøbI &Purity/*FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP9÷JùbE "Purity/2FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP#ïJúbE "Purity/3FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP‹Ì ’NûbI &Purity/*FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP­ºm?;üb6 jet/accuracy_2_of_5"trainer/global_step2:JPIïÉ/NýbI &Purity/*FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPd{” NþbI &Purity/1FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP¸xIHÿbC Purity/1FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP?»ÕvN€ bI &Purity/2FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPõq?3N bI &Purity/2FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPRüäoN‚ bI &Purity/3FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPëÜ>|Hƒ bC Purity/2FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPŠØ»J„ bE "Purity/3FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP˜œè»J… bE "Purity/*FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP Ú±#H† bC Purity/*FRt4SRqqt1FBt/FBt_purity"trainer/global_step2:JPRGû÷J‡ bE "Purity/0FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPƒÊiùHˆ bC Purity/3FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JP ÆøN‰ bI &Purity/*FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP²½ÝJŠ bE "Purity/0FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPaŠxJ‹ bE "Purity/1FRt*SRqqt3FBt/event_purity"trainer/global_step2:JP¾ùëHŒ bC Purity/1FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JP¡ÏðJ bE "Purity/1FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP·MjHŽ bC Purity/*FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JP¨ ŒJ bE "Purity/*FRt1SRqqt4FBt/event_purity"trainer/global_step2:JPÁ먢J bE "Purity/4FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPJ%¶H‘ bC Purity/1FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPa ŒON’ bI &Purity/1FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPÐ1¼®H“ bC Purity/2FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPOO‰aN” bI &Purity/3FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPß—rN• bI &Purity/4FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP*&dxH– bC Purity/1FRt0SRqqt1FBt/FBt_purity"trainer/global_step2:JPŠ€GN— bI &Purity/2FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JP EÏžJ˜ bE "Purity/2FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÉZnaN™ bI &Purity/3FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP‹©áNš bI &Purity/4FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPGõeÎJ› bE "Purity/0FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPR¬=ÇJœ bE "Purity/0FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPq*\[J bE "Purity/3FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPmÐä²Nž bI &Purity/*FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP&JŸ bE "Purity/0FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPp‰ïúN  bI &Purity/2FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPƒdstN¡ bI &Purity/2FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPûcŒ¿N¢ bI &Purity/4FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP&6¦;£ b6 jet/accuracy_1_of_1"trainer/global_step2:JP¬·H¤ bC Purity/3FRt0SRqqt1FBt/FRt_purity"trainer/global_step2:JPÕULô;¥ b6 jet/accuracy_1_of_5"trainer/global_step2:JP#ªH¦ bC Purity/*FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JP¸(VaH§ bC Purity/0FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPÂõeJ¨ bE "Purity/1FRt*SRqqt*FBt/event_purity"trainer/global_step2:JP™/ÏÒH© bC Purity/1FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JP¹¨C€Jª bE "Purity/3FRt2SRqqt0FBt/event_purity"trainer/global_step2:JP{hÊN« bI &Purity/3FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JP#54 H¬ bC Purity/1FRt0SRqqt4FBt/FRt_purity"trainer/global_step2:JP¿•cH­ bC Purity/1FRt*SRqqt2FBt/FRt_purity"trainer/global_step2:JPŒ(g¸N® bI &Purity/4FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPäµõ;¯ b6 jet/accuracy_4_of_5"trainer/global_step2:JPÖº.EH° bC Purity/1FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JP¤ØãÔH± bC Purity/*FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPW¥°N² bI &Purity/*FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP2½'H³ bC Purity/1FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPÝ%—7N´ bI &Purity/1FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JPU蘧;µ b6 jet/accuracy_1_of_4"trainer/global_step2:JPcBJ¶ bE "Purity/3FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP5lq²H· bC Purity/0FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP1ð±J¸ bE "Purity/3FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP.ÿ"\@¹ b; particle/accuracy_4_of_7"trainer/global_step2:JPÄ ÝÈ;º b6 jet/accuracy_6_of_6"trainer/global_step2:JPÖÎ H» bC Purity/1FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JPÓ@ÇÌH¼ bC Purity/*FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JPí!bEJ½ bE "Purity/0FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPÖ8fƒN¾ bI &Purity/1FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JP}À|…N¿ bI &Purity/3FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÍŠ[µNÀ bI &Purity/3FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPÖbØHÁ bC Purity/*FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JP¦ÍßH bC Purity/1FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JP,@Ùy/0FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP_ZæHî bC Purity/1FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JPûÐÎJï bE "Purity/1FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¡“€@ð b; particle/accuracy_4_of_5"trainer/global_step2:JPBãx‰Hñ bC Purity/0FRt4SRqqt*FBt/FBt_purity"trainer/global_step2:JP¢ÃwHò bC Purity/1FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPA²)vHó bC Purity/2FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPòO½Nô bI &Purity/2FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPàŒÛsNõ bI &Purity/2FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPœ9ÿHö bC Purity/*FRt*SRqqt4FBt/FBt_purity"trainer/global_step2:JPý%;÷ b6 jet/accuracy_7_of_7"trainer/global_step2:JP‘*ŸjJø bE "Purity/1FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPôc7²Jù bE "Purity/1FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPª“+™Jú bE "Purity/1FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPK©ŒãNû bI &Purity/4FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPˆŠ4îJü bE "Purity/3FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPFR{;Jý bE "Purity/4FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPN•CÌJþ bE "Purity/1FRt0SRqqt*FBt/event_purity"trainer/global_step2:JP€#æuHÿ bC Purity/1FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP×0õN€ bI &Purity/1FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JP³4?cJ bE "Purity/2FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPÊŠ'RJ‚ bE "Purity/2FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP&CÊ•Jƒ bE "Purity/0FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPD´J„ bE "Purity/1FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPÑ\СJ… bE "Purity/4FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP-9YN† bI &Purity/*FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPш8šN‡ bI &Purity/*FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPw<Ó7Hˆ bC Purity/*FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JP¨ÉÀÎH‰ bC Purity/*FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPŒTdGNŠ bI &Purity/0FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JP?_Ù³H‹ bC Purity/2FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JPš²Ñ@Œ b; particle/accuracy_2_of_6"trainer/global_step2:JP•¡sD@ b; particle/accuracy_6_of_7"trainer/global_step2:JPvuS¨HŽ bC Purity/*FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JPšÄ J bE "Purity/1FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP£uBN bI &Purity/4FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JPài¡²H‘ bC Purity/*FRt1SRqqt4FBt/FRt_purity"trainer/global_step2:JP‹#dJ’ bE "Purity/*FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP‰å—,J“ bE "Purity/*FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP°~5ÝJ” bE "Purity/0FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¹MJ• bE "Purity/1FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP»ò N– bI &Purity/4FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JPž(ãœ@— b; particle/accuracy_5_of_5"trainer/global_step2:JP€ÁìÍH˜ bC Purity/3FRt0SRqqt*FBt/FBt_purity"trainer/global_step2:JPT늯H™ bC Purity/0FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPTŽEíHš bC Purity/*FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPÛ£ HJ› bE "Purity/1FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP@ïùJœ bE "Purity/2FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP‚½ô; b6 jet/accuracy_4_of_4"trainer/global_step2:JP­ªJž bE "Purity/0FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP¨´†)HŸ bC Purity/1FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JP©¦·±N  bI &Purity/2FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPaœ¿îN¡ bI &Purity/2FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JPnþSmJ¢ bE "Purity/0FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP9yÕJ£ bE "Purity/0FRt3SRqqt0FBt/event_purity"trainer/global_step2:JP@ßdJ¤ bE "Purity/3FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÜ­BüN¥ bI &Purity/*FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPn­†ÈJ¦ bE "Purity/0FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPaµê¼J§ bE "Purity/0FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP|¦ÝH¨ bC Purity/1FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPc¦§ÂN© bI &Purity/3FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPò\VNª bI &Purity/4FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JPï'—¸H« bC Purity/2FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JP5+² J¬ bE "Purity/1FRt3SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPV³4iN­ bI &Purity/4FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP§ÅµH® bC Purity/1FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPÂ!N¯ bI &Purity/2FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JPQ .N° bI &Purity/4FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP@@¥;± b6 jet/accuracy_1_of_2"trainer/global_step2:JP—u¿H² bC Purity/4FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JP:xÑòJ³ bE "Purity/1FRt4SRqqt0FBt/event_purity"trainer/global_step2:JP^¤ N´ bI &Purity/*FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JPÿb]—Jµ bE "Purity/0FRt0SRqqt1FBt/event_purity"trainer/global_step2:JPþ)ðN¶ bI &Purity/1FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPx'H’N· bI &Purity/2FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPöœv’H¸ bC Purity/*FRt3SRqqt1FBt/FBt_purity"trainer/global_step2:JP!?º¡J¹ bE "Purity/0FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPâ‰ÙðHº bC Purity/*FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP=|Ê H» bC Purity/*FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JP>‡–N¼ bI &Purity/2FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPÀF èJ½ bE "Purity/3FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPé§N¾ bI &Purity/4FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPË Ô(;¿ b6 jet/accuracy_2_of_2"trainer/global_step2:JPR ?JÀ bE "Purity/3FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPEI` JÁ bE "Purity/2FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP©Ð N bI &Purity/*FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPœ¦ô5@à b; particle/accuracy_2_of_4"trainer/global_step2:JPVî–7@Ä b; particle/accuracy_3_of_5"trainer/global_step2:JPb%ñHÅ bC Purity/4FRt0SRqqt0FBt/FRt_purity"trainer/global_step2:JPàe¬JÆ bE "Purity/0FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP© 0 NÇ bI &Purity/1FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPúÓÚANÈ bI &Purity/0FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPð2™NÉ bI &Purity/4FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPZã£)JÊ bE "Purity/2FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP öØUJË bE "Purity/*FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPÐCQoJÌ bE "Purity/*FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¬Ú°`NÍ bI &Purity/1FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPÁD@JÎ bE "Purity/1FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPmx*°NÏ bI &Purity/1FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP…nË JÐ bE "Purity/2FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP¼óíHÑ bC Purity/2FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP+ 0šNÒ bI &Purity/2FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP©ÒÁPNÓ bI &Purity/4FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JP';NÔ bI &Purity/1FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPáÛÑJÕ bE "Purity/1FRt1SRqqt0FBt/event_purity"trainer/global_step2:JP? AaNÖ bI &Purity/1FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP!#[ÞJ× bE "Purity/2FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP<,»HØ bC Purity/2FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPêw+kNÙ bI &Purity/2FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPš¬¯¯NÚ bI &Purity/4FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JP ÿÃ:HÛ bC Purity/1FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JP6áÈÑNÜ bI &Purity/1FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP‰§NÝ bI &Purity/2FRt3SRqqt3FBt/event_proportion"trainer/global_step2:JPœ²ÓÚNÞ bI &Purity/3FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPÇPO>Hß bC Purity/3FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JP–âWJà bE "Purity/*FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPDûnµNá bI &Purity/4FRt4SRqqt0FBt/event_proportion"trainer/global_step2:JP¾N¢²;â b6 jet/accuracy_3_of_3"trainer/global_step2:JPL¥ú:Jã bE "Purity/*FRt2SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP{Ã&=Hä bC Purity/*FRt3SRqqt2FBt/FRt_purity"trainer/global_step2:JP÷[ZJå bE "Purity/*FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPäCÛEHæ bC Purity/2FRt3SRqqt*FBt/FRt_purity"trainer/global_step2:JPÔoÍÜNç bI &Purity/3FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JP ÂNè bI &Purity/3FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPÇÏñJé bE "Purity/3FRt1SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP#HÎ);ê b6 jet/accuracy_4_of_7"trainer/global_step2:JPY–$'Hë bC Purity/3FRt1SRqqt2FBt/FBt_purity"trainer/global_step2:JPa“ÅJì bE "Purity/3FRt3SRqqt*FBt/event_purity"trainer/global_step2:JPö³ÒœNí bI &Purity/*FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPd!<&Jî bE "Purity/2FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPŠ†R§Hï bC Purity/*FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPe‡V­Hð bC Purity/1FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JPÈ«•ÖJñ bE "Purity/4FRt1SRqqt*FBt/event_purity"trainer/global_step2:JPáߨDNò bI &Purity/2FRt1SRqqt2FBt/event_proportion"trainer/global_step2:JPÅÅ¥ÕJó bE "Purity/0FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP6ݬHô bC Purity/3FRt0SRqqt*FBt/FRt_purity"trainer/global_step2:JPÀÝý´Nõ bI &Purity/3FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP—ÉNö bI &Purity/4FRt*SRqqt3FBt/event_proportion"trainer/global_step2:JPòèýŒN÷ bI &Purity/*FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPIYCîNø bI &Purity/1FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPžC‘Jù bE "Purity/1FRt3SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPèåbo@ú b; particle/accuracy_2_of_3"trainer/global_step2:JP,Ý$Jû bE "Purity/2FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPFÕ·ÎJü bE "Purity/0FRt*SRqqt*FBt/event_purity"trainer/global_step2:JPã38†Hý bC Purity/0FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JPzøšJþ bE "Purity/1FRt*SRqqt2FBt/SRqqt_purity"trainer/global_step2:JP¼+½Nÿ bI &Purity/1FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP˜%xFH€ bC Purity/2FRt3SRqqt0FBt/FRt_purity"trainer/global_step2:JPØshJ bE "Purity/2FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPø²0H‚ bC Purity/3FRt0SRqqt2FBt/FRt_purity"trainer/global_step2:JPÕHƒ bC Purity/*FRt0SRqqt2FBt/FBt_purity"trainer/global_step2:JP'˵N„ bI &Purity/1FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JPÏ7ÅóJ… bE "Purity/2FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP¼Ï9ÒN† bI &Purity/3FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP³°„éJ‡ bE "Purity/*FRt*SRqqt1FBt/event_purity"trainer/global_step2:JP}c¿~Nˆ bI &Purity/0FRt*SRqqt0FBt/event_proportion"trainer/global_step2:JPðŽLJØ bE "Purity/*FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP'*¡ŒJÙ bE "Purity/0FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP“îÚ¬JÚ bE "Purity/1FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP«äËHÛ bC Purity/*FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JP­%9,JÜ bE "Purity/*FRt0SRqqt2FBt/event_purity"trainer/global_step2:JPõ¤bHÝ bC Purity/2FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPà¿Á­HÞ bC Purity/*FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPzß4Nß bI &Purity/*FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JP¦–Q‚Nà bI &Purity/0FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JP*¶žðHá bC Purity/0FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JP4¯MŒJâ bE "Purity/1FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP9§o @ã b; particle/accuracy_4_of_6"trainer/global_step2:JPZÒ{˜Nä bI &Purity/1FRt3SRqqt*FBt/event_proportion"trainer/global_step2:JP„$û½Jå bE "Purity/0FRt0SRqqt3FBt/event_purity"trainer/global_step2:JP½ÿ$ÞHæ bC Purity/1FRt3SRqqt1FBt/FRt_purity"trainer/global_step2:JPmØ’Hç bC Purity/3FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPßQppHè bC Purity/1FRt1SRqqt3FBt/FRt_purity"trainer/global_step2:JPøÁ»ñJé bE "Purity/4FRt2SRqqt*FBt/event_purity"trainer/global_step2:JPÊšNê bI &Purity/*FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPi=SJë bE "Purity/*FRt*SRqqt3FBt/event_purity"trainer/global_step2:JPÀKöNì bI &Purity/3FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPþ¾+OJí bE "Purity/0FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPNøàñHî bC Purity/*FRt1SRqqt4FBt/FBt_purity"trainer/global_step2:JPY,nèHï bC Purity/2FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPR^°Nð bI &Purity/1FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JP€n¦xHñ bC Purity/1FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JPGÌXl;ò b6 jet/accuracy_2_of_4"trainer/global_step2:JPöq¯ Jó bE "Purity/1FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPÙÄPJô bE "Purity/*FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP4È~Jõ bE "Purity/1FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP”uMNö bI &Purity/1FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPĸJ÷ bE "Purity/2FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP¡ ÅNø bI &Purity/4FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP¶ÚÍDJù bE "Purity/1FRt3SRqqt1FBt/event_purity"trainer/global_step2:JP hLoHú bC Purity/2FRt4SRqqt*FBt/FRt_purity"trainer/global_step2:JPèØ¿çHû bC Purity/3FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JPÛ‡ÁNü bI &Purity/4FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPþaQ;ý b6 jet/accuracy_2_of_7"trainer/global_step2:JPb™”Jþ bE "Purity/0FRt2SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPЗëJÿ bE "Purity/4FRt0SRqqt*FBt/event_purity"trainer/global_step2:JPs  N€ bI &Purity/3FRt3SRqqt1FBt/event_proportion"trainer/global_step2:JPej&(J bE "Purity/3FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP‡ šîJ‚ bE "Purity/*FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP]–ÒUNƒ bI &Purity/*FRt2SRqqt*FBt/event_proportion"trainer/global_step2:JPcOÜÞN„ bI &Purity/0FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPMVþ”N… bI &Purity/2FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP­?mÇN† bI &Purity/2FRt3SRqqt0FBt/event_proportion"trainer/global_step2:JP®çL N‡ bI &Purity/4FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP`;oêJˆ bE "Purity/1FRt*SRqqt4FBt/event_purity"trainer/global_step2:JP¿$ûJ‰ bE "Purity/*FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPöC­CNŠ bI &Purity/4FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP‚q…N‹ bI &Purity/4FRt4SRqqt4FBt/event_proportion"trainer/global_step2:JPÝ×ø2@Œ b; particle/accuracy_6_of_6"trainer/global_step2:JPÌðR•@ b; particle/accuracy_5_of_7"trainer/global_step2:JPËl?—JŽ bE "Purity/0FRt2SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP©JlèJ bE "Purity/1FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP‘ö£gJ bE "Purity/*FRt3SRqqt2FBt/event_purity"trainer/global_step2:JP}HºJN‘ bI &Purity/0FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPí$ÚN’ bI &Purity/1FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPº£àJ“ bE "Purity/1FRt1SRqqt2FBt/event_purity"trainer/global_step2:JPŸ—pN” bI &Purity/1FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JP€²N• bI &Purity/3FRt2SRqqt1FBt/event_proportion"trainer/global_step2:JP‹ÑºøN– bI &Purity/3FRt4SRqqt1FBt/event_proportion"trainer/global_step2:JP í@— b; particle/accuracy_1_of_1"trainer/global_step2:JP’4BßN˜ bI &Purity/0FRt*SRqqt*FBt/event_proportion"trainer/global_step2:JPáªJ™ bE "Purity/1FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPPÓ`(Nš bI &Purity/2FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPÀþIóJ› bE "Purity/*FRt4SRqqt*FBt/event_purity"trainer/global_step2:JP/‰RHœ bC Purity/*FRt2SRqqt2FBt/FRt_purity"trainer/global_step2:JP|qtâ@ b; particle/accuracy_3_of_7"trainer/global_step2:JP5è‘"Nž bI &Purity/*FRt2SRqqt3FBt/event_proportion"trainer/global_step2:JPTœäJŸ bE "Purity/2FRt*SRqqt2FBt/event_purity"trainer/global_step2:JPÓÐîqN  bI &Purity/3FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JP¦ízH¡ bC Purity/1FRt1SRqqt3FBt/FBt_purity"trainer/global_step2:JPå!xYH¢ bC Purity/*FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPÆz2ZH£ bC Purity/0FRt2SRqqt1FBt/FBt_purity"trainer/global_step2:JPäZµH¤ bC Purity/2FRt1SRqqt1FBt/FRt_purity"trainer/global_step2:JP—Iñ—N¥ bI &Purity/3FRt2SRqqt4FBt/event_proportion"trainer/global_step2:JPêt@¦ b; particle/accuracy_3_of_3"trainer/global_step2:JPºùÊJ§ bE "Purity/1FRt3SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP×æÆJ¨ bE "Purity/3FRt1SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP¡6?‡@© b; particle/accuracy_7_of_7"trainer/global_step2:JP“×#KNª bI &Purity/*FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JPº’½ÊH« bC Purity/*FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPÍŸõ@¬ b; particle/accuracy_1_of_6"trainer/global_step2:JP}±›lJ­ bE "Purity/3FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP/þuJ® bE "Purity/1FRt3SRqqt2FBt/event_purity"trainer/global_step2:JPцJ¯ bE "Purity/*FRt2SRqqt3FBt/event_purity"trainer/global_step2:JPÜ:ªËN° bI &Purity/3FRt1SRqqt4FBt/event_proportion"trainer/global_step2:JP5Ó—H± bC Purity/0FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JP¦sŸJ² bE "Purity/*FRt2SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPY-­};³ b6 jet/accuracy_5_of_5"trainer/global_step2:JPÊ%´@´ b; particle/accuracy_2_of_2"trainer/global_step2:JPYµ+ì@µ b; particle/accuracy_1_of_4"trainer/global_step2:JP&zUzH¶ bC Purity/2FRt*SRqqt3FBt/FRt_purity"trainer/global_step2:JPpµÞJ· bE "Purity/*FRt1SRqqt4FBt/SRqqt_purity"trainer/global_step2:JPÈL`J¸ bE "Purity/0FRt2SRqqt3FBt/event_purity"trainer/global_step2:JPûsÛ)H¹ bC Purity/*FRt*SRqqt3FBt/FBt_purity"trainer/global_step2:JPM›7ªNº bI &Purity/0FRt0SRqqt4FBt/event_proportion"trainer/global_step2:JPƒ>ž*J» bE "Purity/1FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JP¬ù‡N¼ bI &Purity/1FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPÒbÄN½ bI &Purity/4FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPRu*Z@¾ b; particle/accuracy_4_of_4"trainer/global_step2:JP8Ó\ <¿ b7 particle/specificity"trainer/global_step2:JP&6d•;À b6 jet/accuracy_1_of_7"trainer/global_step2:JP0‹\oHÁ bC Purity/2FRt*SRqqt1FBt/FRt_purity"trainer/global_step2:JPTÿ\xN bI &Purity/3FRt1SRqqt1FBt/event_proportion"trainer/global_step2:JPDíŠJà bE "Purity/*FRt4SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPZÑË_JÄ bE "Purity/2FRt4SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP6ÈJÅ bE "Purity/*FRt1SRqqt3FBt/event_purity"trainer/global_step2:JP뼸‹JÆ bE "Purity/2FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPèÏîHÇ bC Purity/*FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPR¿8JÈ bE "Purity/2FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP6 v;É b6 jet/accuracy_1_of_3"trainer/global_step2:JPuE»<Ê b7 particle/sensitivity"trainer/global_step2:JPËBÿúHË bC Purity/1FRt2SRqqt2FBt/FBt_purity"trainer/global_step2:JPEAJHÌ bC Purity/*FRt2SRqqt*FBt/FRt_purity"trainer/global_step2:JP?]è|NÍ bI &Purity/0FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JPÍLñþJÎ bE "Purity/0FRt0SRqqt0FBt/event_purity"trainer/global_step2:JP!û.rNÏ bI &Purity/2FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JPž¯u’HÐ bC Purity/0FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JPØS©ûJÑ bE "Purity/3FRt0SRqqt1FBt/event_purity"trainer/global_step2:JP]*iJÒ bE "Purity/3FRt*SRqqt0FBt/event_purity"trainer/global_step2:JP@þ;@Ó b; particle/accuracy_1_of_3"trainer/global_step2:JPÜ),JÔ bE "Purity/*FRt*SRqqt3FBt/SRqqt_purity"trainer/global_step2:JP< 6¾JÕ bE "Purity/*FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPÞ8@HÖ bC Purity/1FRt*SRqqt2FBt/FBt_purity"trainer/global_step2:JP+WµDJ× bE "Purity/2FRt*SRqqt*FBt/SRqqt_purity"trainer/global_step2:JP&¬½NØ bI &Purity/2FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPÙ”ÞuJÙ bE "Purity/*FRt2SRqqt2FBt/event_purity"trainer/global_step2:JP,ã‚HÚ bC Purity/2FRt0SRqqt3FBt/FBt_purity"trainer/global_step2:JP£8ÄKJÛ bE "Purity/3FRt0SRqqt2FBt/event_purity"trainer/global_step2:JP„†cJÜ bE "Purity/0FRt*SRqqt0FBt/event_purity"trainer/global_step2:JPuœÍNÝ bI &Purity/1FRt3SRqqt4FBt/event_proportion"trainer/global_step2:JPã5>°HÞ bC Purity/2FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPâfœ$Hß bC Purity/2FRt1SRqqt2FBt/FRt_purity"trainer/global_step2:JPB ]‹;à b6 jet/accuracy_3_of_4"trainer/global_step2:JPMëf‡@á b; particle/accuracy_3_of_6"trainer/global_step2:JP—«lHâ bC Purity/1FRt4SRqqt0FBt/FRt_purity"trainer/global_step2:JPgÍqJã bE "Purity/0FRt1SRqqt3FBt/SRqqt_purity"trainer/global_step2:JPj/ŠNä bI &Purity/1FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPÛ™Nå bI &Purity/3FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPPÜO~Jæ bE "Purity/3FRt*SRqqt1FBt/SRqqt_purity"trainer/global_step2:JPWX—Hç bC Purity/*FRt3SRqqt2FBt/FBt_purity"trainer/global_step2:JPìè;è b6 loss/val_total_loss"trainer/global_step2:JPù÷ äNé bI &Purity/*FRt2SRqqt0FBt/event_proportion"trainer/global_step2:JP{€6¼Nê bI &Purity/2FRt*SRqqt4FBt/event_proportion"trainer/global_step2:JP.j[Jë bE "Purity/*FRt4SRqqt0FBt/event_purity"trainer/global_step2:JPHÓHì bC Purity/3FRt2SRqqt*FBt/FBt_purity"trainer/global_step2:JPÿzJí bE "Purity/*FRt1SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPŸ½Nî bI &Purity/*FRt3SRqqt2FBt/event_proportion"trainer/global_step2:JP„{ÂJï bE "Purity/1FRt*SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPƒEAœJð bE "Purity/2FRt3SRqqt0FBt/event_purity"trainer/global_step2:JPiüNñ bI &Purity/2FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPáS\ Hò bC Purity/4FRt*SRqqt0FBt/FRt_purity"trainer/global_step2:JPÙ[ÇHó bC Purity/4FRt2SRqqt0FBt/FRt_purity"trainer/global_step2:JPTÉl.Nô bI &Purity/4FRt2SRqqt2FBt/event_proportion"trainer/global_step2:JPAl§÷;õ b6 jet/accuracy_2_of_3"trainer/global_step2:JPÿL>ŸHö bC Purity/1FRt0SRqqt4FBt/FBt_purity"trainer/global_step2:JPz®˜;H÷ bC Purity/4FRt1SRqqt*FBt/FRt_purity"trainer/global_step2:JP €3ÁNø bI &Purity/*FRt0SRqqt3FBt/event_proportion"trainer/global_step2:JPB«áÈNù bI &Purity/*FRt1SRqqt0FBt/event_proportion"trainer/global_step2:JPE±CñHú bC Purity/1FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP-ndÇHû bC Purity/2FRt1SRqqt0FBt/FRt_purity"trainer/global_step2:JPþ/™Jü bE "Purity/2FRt1SRqqt1FBt/event_purity"trainer/global_step2:JPó6GJý bE "Purity/2FRt1SRqqt2FBt/SRqqt_purity"trainer/global_step2:JPÇmHJþ bE "Purity/2FRt2SRqqt0FBt/event_purity"trainer/global_step2:JPWN’ @ÿ b; particle/accuracy_5_of_6"trainer/global_step2:JP$m´:H€ bC Purity/*FRt2SRqqt1FBt/FRt_purity"trainer/global_step2:JP¼¶¼ïJ bE "Purity/*FRt3SRqqt0FBt/SRqqt_purity"trainer/global_step2:JP˜UÉN‚ bI &Purity/*FRt4SRqqt*FBt/event_proportion"trainer/global_step2:JPø¥p0Jƒ bE "Purity/0FRt*SRqqt2FBt/event_purity"trainer/global_step2:JP§ çÿ3„ bI &Purity/1FRt2SRqqt*FBt/event_proportion"trai+ʼnÌner/global_step2:JP·Í­cH… bC Purity/2FRt1SRqqt1FBt/FBt_purity"trainer/global_step2:JP9?föN† bI &Purity/3FRt*SRqqt1FBt/event_proportion"trainer/global_step2:JPzÀU‚N‡ bI &Purity/3FRt0SRqqt1FBt/event_proportion"trainer/global_step2:JPCÖJˆ bE "Purity/*FRt2SRqqt0FBt/SRqqt_purity"trainer/global_step2:JPêSN‰ bI &Purity/0FRt4SRqqt2FBt/event_proportion"trainer/global_step2:JP$³^NŠ bI &Purity/3FRt0SRqqt*FBt/event_proportion"trainer/global_step2:JP¶aB“N‹ bI &Purity/4FRt0SRqqt2FBt/event_proportion"trainer/global_step2:JP;iT'NŒ bI &Purity/4FRt4SRqqt3FBt/event_proportion"trainer/global_step2:JPìòÅH bC Purity/1FRt3SRqqt*FBt/FBt_purity"trainer/global_step2:JPßÛúæHŽ bC Purity/3FRt1SRqqt*FBt/FBt_purity"trainer/global_step2:JP0 ^®J bE "Purity/1FRt2SRqqt2FBt/event_purity"trainer/global_step2:JPVÍ*J bE "Purity/*FRt3SRqqt*FBt/event_purity"trainer/global_step2:JP“lý@N‘ bI &Purity/0FRt1SRqqt3FBt/event_proportion"trainer/global_step2:JPÙZ@N’ bI &Purity/1FRt1SRqqt*FBt/event_proportion"trainer/global_step2:JPýÄ H“ bC Purity/2FRt*SRqqt1FBt/FBt_purity"trainer/global_step2:JPý´ÉN” bI &Purity/3FRt*SRqqt2FBt/event_proportion"trainer/global_step2:JP#¸õJ• bE "Purity/0FRt4SRqqt*FBt/SRqqt_purity"trainer/global_step2:JPé%fcJ– bE "Purity/1FRt0SRqqt4FBt/event_purity"trainer/global_step2:JPäPļH— bC Purity/4FRt*SRqqt*FBt/FRt_purity"trainer/global_step2:JPÙûÚSH˜ bC Purity/0FRt*SRqqt*FBt/FBt_purity"trainer/global_step2:JPÆÒ:¾N™ bI &Purity/1FRt0SRqqt0FBt/event_proportion"trainer/global_step2:JPœ*‹;@š b; particle/accuracy_2_of_5"trainer/global_step2:JPݘ›lJ› bE "Purity/3FRt4SRqqt*FBt/event_purity"trainer/global_step2:JPiåfª;œ b6 jet/accuracy_3_of_7"trainer/global_step2:JP¿˜Ê«0 + _wandbruntime‚985  _runtime‚985C›œGxž ‡¦ A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.375 jet/accuracy_4_of_7‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 trainer/global_step‚1208 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3931650221347809 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.10185185074806213 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.24521073698997498 particle/accuracy_2_of_3‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3460824191570282 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9759381413459778 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3790293335914612 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 6 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.212373748421669 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9778416156768799 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.35438627004623413 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.009433962404727936 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.04988479241728783 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 *jet/accuracy_1_of_6‚0.9487180113792419 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7883963610195355e+09 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3789920210838318 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 7validation_average_jet_accuracy‚0.31847289204597473 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.34375 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9697859883308411 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.004176114685833454 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.24755288660526276 particle/accuracy_1_of_2‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.05303030088543892 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.053333476185798645 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9710643291473389 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9766190648078918 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.09730818122625351 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.24020059406757355 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.22163715958595276 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.004629630129784346 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.1971726268529892 (particle/accuracy‚0.8085781335830688 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.3529411852359772 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.23963934183120728 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.18425928056240082 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17849352955818176 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.2329082489013672 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.2697649598121643 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.0472051165997982 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.26934298872947693 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.005442176945507526 +jet/accuracy_3_of_6‚0.11538461595773697 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.21804583072662354 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.22264167666435242 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9803181290626526 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.39609718322753906 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8182188272476196 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04197454825043678 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9571018218994141 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.21034854650497437 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.34210526943206787 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2371329367160797 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9797487258911133 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9904761910438538 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.036373529583215714 particle/accuracy_4_of_6‚1 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.15625 ) Purity/1FRt3SRqqt1FBt/FRt_purity‚0.25 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12411347776651382 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2174757421016693 +jet/accuracy_2_of_4‚0.21739189326763153 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6226478815078735 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.23226124048233032 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.22715985774993896 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.07846492528915405 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 particle/accuracy_5_of_7‚1 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.250282883644104 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.06276693195104599 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.21148476004600525 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 particle/accuracy_3_of_7‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9845994710922241 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9924528002738953 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.26324406266212463 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.34375 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.39120373129844666 particle/accuracy_7_of_7‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12876583635807037 particle/accuracy_1_of_6‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 _runtime‚ 985.658968098 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991228342056274 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37836989760398865 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_4‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9160757064819336 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.39710792899131775 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 +particle/specificity‚0.9151970148086548 jet/accuracy_1_of_7‚0.5 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.23071788251399994 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.001831501955166459 *jet/accuracy_1_of_3‚0.7084174752235413 ,particle/sensitivity‚0.23224757611751556 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9166666865348816 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1344228982925415 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 - Purity/0FRt0SRqqt3FBt/FBt_purity‚0.921875 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.699999988079071 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.05322639271616936 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9631655812263489 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.2527221441268921 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.17961879074573517 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.13917148113250732 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.21590909361839294 ,jet/accuracy_3_of_4‚0.023869957774877548 particle/accuracy_3_of_6‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5000000596046448 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 )loss/val_total_loss‚64.78367497656551 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.22230194509029388 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.21301494538784027 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.1432807743549347 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.974770724773407 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.11210259795188904 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5757575631141663 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.17594628036022186 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.20811358094215393 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27850663661956787 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.21949456632137299 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9968253970146179 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9802900552749634 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 particle/accuracy_2_of_5‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_3_of_7‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_6‚0.32692307233810425 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9819263815879822 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9592325687408447 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.10499206185340881 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9842739701271057 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.2746913433074951 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9860624074935913 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.22717873752117157 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.2538760006427765 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9952381253242493 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1487230658531189 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3710317611694336 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.09959550946950912 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0029986060690134764 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3024691045284271 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.28394168615341187 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.997551441192627 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16289232671260834 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.07349057495594025 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 +jet/accuracy_4_of_6‚0.01923076994717121 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9978256821632385 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.10991314798593521 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1425287276506424 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.12037037312984467 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 _step‚98 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.06588084995746613 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0013938669580966234 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7"Purity/*FRt0SRqqt3FBt/event_purity‚0.15079365670681 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9917941689491272 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.06666667014360428 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333134651184 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.10790547728538513 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9955146312713623 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.2785087525844574 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.2291666716337204 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9920490980148315 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.3044223189353943 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14080460369586945 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0035714285913854837 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.12979277968406677 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.984628438949585 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.023969534784555435 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.14941519498825073 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.05111111328005791 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.13800418376922607 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14519378542900085 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.12298054993152618 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1491769552230835 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.20990243554115295 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.32380956411361694 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.017518721520900726 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.14289329946041107 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.22697632014751434 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.22256942093372345 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2152777761220932 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.11814296990633011 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.027803437784314156 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.08918704837560654 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12646761536598206 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.1792929619550705 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3214285671710968 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2083333283662796 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2222222238779068 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_7‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2291666716337204 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.2582201361656189 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.007111619226634502 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2263769656419754 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 (particle/f_score‚0.27465513348579407 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.007933666929602623 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.13239555060863495 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.34710994362831116 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.20870812237262726 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_5‚0.27995389699935913 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9625991582870483 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.407853364944458 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2291666716337204 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.19004642963409424 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.3367941379547119 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2083333283662796 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9806749224662781 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1398274004459381 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9977324604988098 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.16741743683815002 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1184210553765297 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.019661646336317062 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.33807721734046936 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.1770833283662796 )jet/accuracy_1_of_5‚0.876017689704895 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9857242703437805 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.03354373946785927 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.06276693195104599 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.2975972294807434 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.002016128972172737 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.23048752546310425 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9876758456230164 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.7939638495445251 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.35333335399627686 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9745674729347229 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.31609195470809937 particle/accuracy_4_of_7‚1 jet/accuracy_6_of_6‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.26060110330581665 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.10927700251340866 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9706896543502808 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3778046667575836 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9900306463241577 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13226422667503357 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.36666667461395264 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.25455406308174133 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2666666805744171 ("Purity/0FRt4SRqqt1FBt/SRqqt_purit6û{Ò y‚0 jet/accuracy_6_of_7‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9300000071525574 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3139997720718384 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.22697632014751434 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1369233876466751 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.37083330750465393 particle/accuracy_4_of_5‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.14571663737297058 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_7_of_7‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.04971376433968544 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2291666716337204 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.07114777714014053 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.21753810346126556 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13188548386096954 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1792098879814148 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 particle/accuracy_2_of_6‚1 particle/accuracy_6_of_7‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9873045086860657 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.34513887763023376 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.38373929262161255 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.38438907265663147 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.25548771023750305 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.208289235830307 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.41970452666282654 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5757575631141663 -jet/accuracy_4_of_4‚0.0009920635493472219 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6137617230415344 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.22011354565620422 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2684539258480072 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.38277605175971985 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.18160918354988098 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3333333432674408 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_2‚0.5895923376083374 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9978256821632385 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.157035231590271 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18188908696174622 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.3081863224506378 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.06302955001592636 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 particle/accuracy_3_of_5‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.35489189624786377 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 epoch‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.16389460861682892 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.24744710326194763 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.002779694739729166 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.2669660151004791 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9619792103767395 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.405600368976593 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.008381386287510395 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3799999952316284 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.22072072327136993 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2198198139667511 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0éÕŠšédŸ ‹¦ ;"Purity/*FRt0SRqqt0FBt/event_purity‚0.053333476185798645 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9710643291473389 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9766190648078918 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.09730818122625351 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.24020059406757355 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.22163715958595276 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.004629630129784346 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.1971726268529892 (particle/accuracy‚0.8085781335830688 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.3529411852359772 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.23963934183120728 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.18425928056240082 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.17849352955818176 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.2329082489013672 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.2697649598121643 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.0472051165997982 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.26934298872947693 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.005442176945507526 +jet/accuracy_3_of_6‚0.11538461595773697 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.21804583072662354 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.22264167666435242 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9803181290626526 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.39609718322753906 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8182188272476196 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.04197454825043678 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9571018218994141 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.21034854650497437 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.34210526943206787 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2371329367160797 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9797487258911133 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9904761910438538 ;"Purity/1FRt*SRqqt2FBt/event_purity‚0.036373529583215714 particle/accuracy_4_of_6‚1 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.15625 ) Purity/1FRt3SRqqt1FBt/FRt_purity‚0.25 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.12411347776651382 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2174757421016693 +jet/accuracy_2_of_4‚0.21739189326763153 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2083333283662796 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6226478815078735 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.23226124048233032 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.22715985774993896 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.0476190485060215 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.07846492528915405 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 particle/accuracy_5_of_7‚1 8"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.250282883644104 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.06276693195104599 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.21148476004600525 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 particle/accuracy_3_of_7‚1 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9845994710922241 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9924528002738953 8 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.26324406266212463 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 ."Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.34375 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.39120373129844666 particle/accuracy_7_of_7‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12876583635807037 particle/accuracy_1_of_6‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 _runtime‚ 985.658968098 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8991228342056274 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.37836989760398865 jet/accuracy_5_of_5‚0 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_4‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9160757064819336 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.39710792899131775 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 +particle/specificity‚0.9151970148086548 jet/accuracy_1_of_7‚0.5 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.23071788251399994 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9246032238006592 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.001831501955166459 *jet/accuracy_1_of_3‚0.7084174752235413 ,particle/sensitivity‚0.23224757611751556 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9166666865348816 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1344228982925415 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 - Purity/0FRt0SRqqt3FBt/FBt_purity‚0.921875 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.699999988079071 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.05322639271616936 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9631655812263489 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.2527221441268921 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.17961879074573517 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.13917148113250732 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.21590909361839294 ,jet/accuracy_3_of_4‚0.023869957774877548 particle/accuracy_3_of_6‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5000000596046448 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 )loss/val_total_loss‚64.78367497656551 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.22230194509029388 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.21301494538784027 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_2_of_3‚0.1432807743549347 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.974770724773407 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.11210259795188904 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.5757575631141663 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.17594628036022186 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.20811358094215393 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27850663661956787 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.21949456632137299 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9968253970146179 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 +"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9802900552749634 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 particle/accuracy_2_of_5‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 jet/accuracy_3_of_7‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_6‚0.32692307233810425 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2083333283662796 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9819263815879822 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9592325687408447 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.10499206185340881 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9842739701271057 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.2746913433074951 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9860624074935913 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.22717873752117157 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.2538760006427765 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9952381253242493 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1487230658531189 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.3710317611694336 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.09959550946950912 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 <"Purity/1FRt1SRqqt*FBt/event_purity‚0.0029986060690134764 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3024691045284271 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.28394168615341187 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 6 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.997551441192627 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.16289232671260834 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.07349057495594025 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 +jet/accuracy_4_of_6‚0.01923076994717121 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9978256821632385 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.10991314798593521 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1425287276506424 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.12037037312984467 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 _step‚98 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.06588084995746613 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0013938669580966234 7 Purity/2FRt0SRqqt1FBt/FBt_purity‚0.9940476417541504 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 7"Purity/*FRt0SRqqt3FBt/event_purity‚0.15079365670681 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9917941689491272 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.06666667014360428 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3333333134651184 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.10790547728538513 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9955146312713623 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.2785087525844574 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.2291666716337204 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9920490980148315 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.3044223189353943 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14080460369586945 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 <"Purity/2FRt0SRqqt*FBt/event_purity‚0.0035714285913854837 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.12979277968406677 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.984628438949585 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.023969534784555435 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.14941519498825073 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.05111111328005791 & Purity/*FRt2SRqqt3FBt/FBt_purity‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.13800418376922607 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/0FRt2SRqqt3FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14519378542900085 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.12298054993152618 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1491769552230835 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.20990243554115295 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.32380956411361694 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_7‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.017518721520900726 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.0833333358168602 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.14289329946041107 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.22697632014751434 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.22256942093372345 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2152777761220932 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.11814296990633011 ;"Purity/1FRt*SRqqt0FBt/event_purity‚0.027803437784314156 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.08918704837560654 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12646761536598206 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.1792929619550705 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.3214285671710968 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2083333283662796 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2222222238779068 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_7‚0 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.2291666716337204 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.2582201361656189 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.007111619226634502 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.2263769656419754 & Purity/2FRt2SRqqt1FBt/FBt_purity‚1 (particle/f_score‚0.27465513348579407 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.007933666929602623 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.13239555060863495 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.34710994362831116 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.20870812237262726 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_2_of_5‚0.27995389699935913 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9625991582870483 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.407853364944458 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.2291666716337204 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.19004642963409424 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.3367941379547119 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2083333283662796 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9806749224662781 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1398274004459381 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt1FBt/FBt_purity‚0.9977324604988098 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.16741743683815002 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1184210553765297 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ;"Purity/0FRt2SRqqt0FBt/event_purity‚0.019661646336317062 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.33807721734046936 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.1770833283662796 )jet/accuracy_1_of_5‚0.876017689704895 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9857242703437805 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.03354373946785927 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.06276693195104599 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.2975972294807434 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.002016128972172737 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.23048752546310425 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9876758456230164 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.7939638495445251 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.35333335399627686 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9745674729347229 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.31609195470809937 particle/accuracy_4_of_7‚1 jet/accuracy_6_of_6‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.26060110330581665 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity|rHï)./*FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.10927700251340866 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9706896543502808 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.3778046667575836 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9900306463241577 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13226422667503357 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.36666667461395264 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.25455406308174133 particle/accuracy_1_of_5‚1 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2666666805744171 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 jet/accuracy_6_of_7‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 ("Purity/2FRt*SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9300000071525574 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3139997720718384 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.22697632014751434 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1369233876466751 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.37083330750465393 particle/accuracy_4_of_5‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.14571663737297058 & Purity/2FRt2SRqqt*FBt/FBt_purity‚1 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_7_of_7‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.04971376433968544 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.2291666716337204 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.07114777714014053 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.21753810346126556 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13188548386096954 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1792098879814148 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 particle/accuracy_2_of_6‚1 particle/accuracy_6_of_7‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9873045086860657 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.34513887763023376 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 :"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.38373929262161255 :"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.38438907265663147 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.25548771023750305 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.208289235830307 :"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.41970452666282654 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.5757575631141663 -jet/accuracy_4_of_4‚0.0009920635493472219 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6137617230415344 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.22011354565620422 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.2684539258480072 :"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.38277605175971985 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.18160918354988098 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3333333432674408 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_2‚0.5895923376083374 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9978256821632385 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.157035231590271 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18188908696174622 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.3081863224506378 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.06302955001592636 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 particle/accuracy_3_of_5‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.35489189624786377 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 epoch‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.16389460861682892 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.24744710326194763 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.002779694739729166 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.2669660151004791 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9619792103767395 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 8"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.405600368976593 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.008381386287510395 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.3799999952316284 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.22072072327136993 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2198198139667511 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.375 jet/accuracy_4_of_7‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 trainer/global_step‚1208 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.3931650221347809 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.10185185074806213 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.24521073698997498 particle/accuracy_2_of_3‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.1666666716337204 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.3460824191570282 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9759381413459778 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.3790293335914612 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 6 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.212373748421669 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9778416156768799 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/2FRt0SRqqt1FBt/event_purity‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.35438627004623413 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.009433962404727936 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.04988479241728783 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 *jet/accuracy_1_of_6‚0.9487180113792419 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 % _timestamp‚1.7883963610195355e+09 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.3789920210838318 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 7validation_average_jet_accuracy‚0.31847289204597473 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.34375 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9697859883308411 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.004176114685833454 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.24755288660526276 particle/accuracy_1_of_2‚1 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.05303030088543892 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05bTcx’  ‚(:‡ ͆ãÔÓ¹ ® gpu.0.gpu‚30.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚120.799' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1598 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²µ1ê|¡ ‚(:ñ ͆ãÔ‡ëɱ disk.rbd0.out‚0( disk.nvme0c0n1.out‚18162.77001953125' proc.memory.availableMB‚ 1914.734375 disk.dm-1.out‚ 2208.27734375$ disk./.usageGB‚260.7511672973633 disk.drbd2265.in‚0# disk.dm-0.out‚18162.77001953125 disk.nvme0n1.in‚ 334.1484375 network.sent‚ 1.582451e+06 disk.sda.out‚ 2208.27734375 disk.drbd2080.in‚0 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 334.1484375# proc.memory.rssMB‚ 2676.57421875* proc.memory.percent‚16.336512565612793 disk.drbd2265.out‚0 disk.dm-0.in‚ 334.1484375 disk.rbd1.out‚ 3.52978515625& disk.nvme0n1.out‚18162.77001953125 cpu‚25.39823753869492 disk.sda1.out‚0 disk.drbd2266.in‚0* disk./.usagePercent‚17.499229622127967 disk.sda.in‚ 0.01953125 disk.sda2.out‚0$ memory_percent‚88.31338882446289 disk.sda2.in‚0 disk.sda3.in‚ 0.01953125 network.recv‚416819 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.01953125 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 2208.27734375 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.rbd0.in‚0”Séõ¢ ‚(:’ ܆ãÔ¢±”® gpu.0.gpu‚28.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.841' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.568199999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[52¹w£ ‚(:ì ܆ãÔ×›°± proc.cpu.threads‚17 network.recv‚422653 disk.sda3.in‚ 0.01953125 disk.drbd1716.out‚0 disk.drbd2266.out‚0# disk.dm-0.out‚18409.17529296875* disk./.usagePercent‚17.499163060169064! disk.nvme0c0n1.in‚ 334.1484375 disk.dm-1.out‚ 2259.1796875 disk.drbd2265.out‚0) proc.memory.availableMB‚ 1914.66796875 disk.rbd1.out‚ 3.59228515625 disk.sda.in‚ 0.01953125 disk.drbd1716.in‚0 network.sent‚ 1.706155e+06$ disk./.usageGB‚260.7501754760742 disk.sda1.in‚0 disk.sda3.out‚ 2259.1796875 disk.sda.out‚ 2259.1796875 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 334.1484375 disk.dm-1.in‚ 0.01953125 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚18409.17529296875 disk.rbd0.out‚0 proc.memory.rssMB‚2676.625 disk.rbd0.in‚0 disk.dm-0.in‚ 334.1484375& disk.nvme0n1.out‚18409.17529296875 disk.drbd2080.in‚0$ memory_percent‚88.31379413604736* proc.memory.percent‚16.336822509765625 cpu‚25.3931762490623821C…£’¤ ‚(:‡ ë†ãÔ”爮 gpu.0.gpu‚20.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.907' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7814 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŒµÛ7j¥ ‚(:ß ë†ãÔ–û­° disk.drbd2266.in‚0 disk.drbd2080.out‚0* proc.memory.percent‚16.336822509765625 disk.sda.out‚2297.625 disk.dm-1.in‚ 0.01953125 disk.drbd2265.out‚0 disk.sda3.in‚ 0.01953125( disk.nvme0c0n1.out‚18409.42919921875$ disk./.usageGB‚260.7501754760742 disk.drbd1716.out‚0 disk.dm-0.in‚ 334.1484375 network.sent‚ 1.712284e+06 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 334.1484375& disk.nvme0n1.out‚18409.42919921875$ memory_percent‚88.31706047058105 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 334.1484375( proc.memory.availableMB‚ 1914.1328125 network.recv‚426865 disk.dm-1.out‚2297.625 disk.drbd2266.out‚0# disk.dm-0.out‚18409.42919921875 proc.memory.rssMB‚2676.625 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.out‚2297.625 disk.rbd1.out‚ 3.59228515625 cpu‚25.388301494728733 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0* disk./.usagePercent‚17.499163060169064 disk.sda.in‚ 0.01953125„âh52¦ - _wandbruntime‚1020  _runtime‚1020¼4/ȧ  ' lr-AdamW/pg1‚0.00021497944513315718 ' lr-AdamW/pg2‚0.00021497944513315718 trainer/global_step‚1249 % _timestamp‚1.7883963958985505e+09 _runtime‚1020.535443729 _step‚99Dn,f̨ Æ ' lr-AdamW/pg1‚0.00021497944513315718 ' lr-AdamW/pg2‚0.00021497944513315718 trainer/global_step‚1249 % _timestamp‚1.7883963958985505e+09 _runtime‚1020.535443729 _step‚99caNåŽ2© - _wandbruntime‚1020  _runtime‚1020å0Ã'ª ½O WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 trainer/global_step‚1249 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1763229851648 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚33566652765184 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚35287875831808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16507371 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚35287875831808 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37050918681600 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1757977266935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚33566635725824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25278711 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16507373 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1793530736128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25298747 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚35287857828864 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206391296 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚35324612992759 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37387422950400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚35324795153515 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25298747 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41786082 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37050918681600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37051105683456 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1763060852736 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37051105683456 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39180953686528Á Rû£« ÁO _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚33566652765184 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚35287875831808 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚16507371 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚35287875831808 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚37050918681600 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1757977266935 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚33566635725824 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚25278711 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚16507373 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1793530736128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚25298747 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚35287857828864 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206391296 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚35324612992759 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚37387422950400 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚35324795153515 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚25298747 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚41786082 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚37050918681600 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚508 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚37051105683456 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1763060852736 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚37051105683456 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚39180953686528 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 trainer/global_step‚1249 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1763229851648 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040d-Ü.C2¬ - _wandbruntime‚1021  _runtime‚1021k½gíÍ&­ ÇM K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16848135 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37079581184512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16520032 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39211049226240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚101 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37416311958528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16520030 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16848644 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7201804 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚33592578908160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚35315145196544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9646333 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1764453990912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚35315127193600 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1764623003136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9646331 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25318560 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚509 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37416335898624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25298498 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7202311 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 % _timestamp‚1.7883963966513543e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1021.288942501 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39211285835776 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212669440 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16520030 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25298498 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37079768199680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚507 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1759366431479 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41818528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1759531553083 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41818528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1794949937152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41838592ÉÁV Ñ&® ËM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1794737267712 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚41838592 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚25318560 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 trainer/global_step‚1249 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236609536 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚16520030 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚16848644 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7201804 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚33592578908160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚35315145196544 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚9646333 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1764453990912 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚35315127193600 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1764623003136 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚9646331 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚25318560 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚509 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚37416335898624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚25298498 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7202311 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 % _timestamp‚1.7883963966513543e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚1021.288942501 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚39211285835776 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212669440 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚16520030 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚25298498 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚37079768199680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚507 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1759366431479 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚41818528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1759531553083 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚41818528 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1794949937152 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚41838592 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚16848135 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚37079581184512 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚16520032 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚39211049226240 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _step‚101 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚37416311958528 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O\?dµ ‚(:Ù ‰‡ãÔçÀ´± disk.drbd2265.out‚0' proc.memory.availableMB‚ 1914.328125 disk.sda2.in‚0 disk.rbd1.out‚ 3.62451171875 disk.nvme0n1.in‚ 334.1484375# proc.memory.rssMB‚ 2676.62890625 disk.sda3.out‚ 2378.53125! disk.nvme0c0n1.in‚ 334.1484375 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 2378.53125 disk.sda1.in‚0 disk.sda3.in‚ 0.0234375* disk./.usagePercent‚17.499124147023856 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.drbd2265.in‚0 disk.dm-1.in‚ 0.0234375 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda.in‚ 0.0234375 disk.sda.out‚ 2378.53125$ disk.nvme0c0n1.out‚ 18410.0234375" disk.nvme0n1.out‚ 18410.0234375$ memory_percent‚88.31586837768555 cpu‚25.37880839525322 network.sent‚ 1.82619e+06 network.recv‚436776 disk.drbd2266.out‚0* proc.memory.percent‚16.336846351623535 proc.cpu.threads‚17% disk./.usageGB‚260.74959564208984 disk.drbd2080.out‚0 disk.dm-0.in‚ 334.1484375 disk.dm-0.out‚ 18410.0234375 disk.rbd1.in‚0FÿBÛ2¶ - _wandbruntime‚1060  _runtime‚1060ñúµiÉ· à _step‚103 ' lr-AdamW/pg1‚0.00021495032471248058 ' lr-AdamW/pg2‚0.00021495032471248058 trainer/global_step‚1299 % _timestamp‚1.7883964356544101e+09 _runtime‚1060.291313872õê˜U͸ Ç trainer/global_step‚1299 % _timestamp‚1.7883964356544101e+09 _runtime‚1060.291313872 _step‚103 ' lr-AdamW/pg1‚0.00021495032471248058 ' lr-AdamW/pg2‚0.00021495032471248058g£SÏ`2¹ - _wandbruntime‚1060  _runtime‚1060j`„¹Ëº ¾O N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚26271413 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _runtime‚ 1060.29247335 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚36651344068608 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10007804 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7464580 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43411734 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26291449 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚38484245242880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1864500114432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TADeviceStatsMonitor.on_train_batch_start/r@±"ù allocated_bytes.all.freed‚38484058241024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚34863794842624 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1827449085343 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17140321 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚40697333853184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682752 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 % _timestamp‚1.7883964356552215e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚38484058241024 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚36691243927967 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36651326065664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1827614206739 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚26271413 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1832732175360 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚38484245242880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1832901174272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚26291449 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚17140321 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚40697570476032 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚17472916 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚17472384 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚36651326065664 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1832732175360 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 trainer/global_step‚1299 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622848 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1864287431680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17140323 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚43431772 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚17140323 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7464580 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚43411734 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚26291449 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚38484245242880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 1864500114432 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚38484058241024 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚34863794842624 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1827449085343 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17140321 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚40697333853184 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682752 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 % _timestamp‚1.7883964356552215e+09 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚38484058241024 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚36691243927967 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚36651326065664 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1827614206739 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0hïÁ­2¼ - _wandbruntime‚1061  _runtime‚1061!›ZÏ&½ ÉM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1834294325760 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚36678595430400 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚38512720743936 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚34889755064320 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚533 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚1061.027132848 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36678595430400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚34889738024960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚38861959369728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36678613433344 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚40727665313280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1829003371491 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26311262 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26291200 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚36678613433344 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43464244 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26291200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10015183 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17152982 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43444180 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17485752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1834125313536 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26311262 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232415232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17485219 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7470569 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43464244 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1834294325760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 % _timestamp‚1.7883964363897738e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚38512907759104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17152982 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1865705943552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17152980 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚38861983309824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚1061.027132848 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚36678595430400 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚34889738024960 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚38861959369728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚36678613433344 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚40727665313280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1829003371491 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚26311262 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚26291200 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚36678613433344 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚43464244 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚26291200 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10015183 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17152982 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚43444180 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚17485752 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 1834125313536 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚26311262 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232415232 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚17485219 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚7470569 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚43464244 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1834294325760 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 % _timestamp‚1.7883964363897738e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚38512907759104 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17152982 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1865705943552 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17152980 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚38861983309824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B’2Ë - _wandbruntime‚1099  _runtime‚1099†xFÃ'Ì ½O _step‚108 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39917197800448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10376654 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27284151 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40278693832704 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10376656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1902403497984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45037386 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚38014794302464 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1902572496896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18113463 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚39917197800448 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17773273 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17773271 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚39917384802304 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1349 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚38014812305408 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45057424 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622848 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚36160953959424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1902403497984 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚42213912032768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚36160970998784 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1896920903751 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27284151 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27264115 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7736809 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚38058057023931 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚42214148655616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1935218200064݉vŸ="Í ÁO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚27284151 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚27264115 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚7736809 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚38058057023931 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚42214148655616 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 1935218200064 _step‚108 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚39917197800448 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10376654 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚27284151 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚40278693832704 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10376656 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1902403497984 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚45037386 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚38014794302464 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1902572496896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18113463 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚39917197800448 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚17773273 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚17773271 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚39917384802304 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚1349 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚38014812305408 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚45057424 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622848 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚36160953959424 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1902403497984 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚42213912032768 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚36160970998784 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 RDDeviceStatsMonitor.on_g÷vPŠtrain_batch_start/active_bytes.small_pool.peak‚ 410711040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1896920903751l'õ±2Î - _wandbruntime‚1100  _runtime‚11007õ/ Ï&Ï ÉM ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚38042081670144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1903965648384 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17785932 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18126906 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1898310068295 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45089896 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10384031 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚505 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17785932 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17785930 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18126399 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚42244241395712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 trainer/global_step‚1349 _step‚109 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚38042081670144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚39946047318528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40307630721024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚36186914181120 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚38042063667200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208475136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45089896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232415232 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27303964 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1936843089920 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27283902 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27303964 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1936634614784 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7742368 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40307606780928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1903796636160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10384033 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚38085389371019 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17785930 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27283902 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚39945860303360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚38042063667200 % _timestamp‚1.7883964759248102e+09 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚42244473810944 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚38085207210055 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚1100.562370868 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040,úKÝÓ&Ð ÍM QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚7742368 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚40307606780928 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1903796636160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10384033 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚38085389371019 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚17785930 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚27283902 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚39945860303360 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚38042063667200 % _timestamp‚1.7883964759248102e+09 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚42244473810944 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚38085207210055 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚1100.562370868 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚38042081670144 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1903965648384 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚17785932 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18126906 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1898310068295 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚45089896 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10384031 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚505 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚17785932 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚17785930 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18126399 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚42244241395712 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 trainer/global_step‚1349 _step‚109 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚38042081670144 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚39946047318528 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚40307630721024 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚36186914181120 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚38042063667200 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208475136 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚45089896 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232415232 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚27303964 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 1936843089920 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚27283902 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚27303964 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 1936634614784 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224m†WÇ’2Ñ - _wandbruntime‚1100  _runtime‚1100J¿\ÖTÒ Î /loss/FBt3/detection_loss‚11.936997175216675 /loss/FBt4/detection_loss‚1.3811501286029644 /loss/FRt2/assignment_loss‚4.261145883676957 /loss/FRt4/detection_loss‚2.5884445784396903 1loss/SRqqt4/detection_loss‚1.4019721318260734 0loss/FRt4/assignment_loss‚2.8866755144540654 /loss/FBt2/assignment_loss‚0.380422824861593 /loss/FRt3/detection_loss‚2.6123480664359198 0loss/SRqqt3/detection_loss‚2.090966126946516 trainer/global_step‚1349 _runtime‚1100.563873504 0loss/FBt4/assignment_loss‚0.3545848075010023 2loss/SRqqt3/assignment_loss‚1.9276423856269482 0loss/FBt1/assignment_loss‚0.3615067465496395 0loss/FBt3/assignment_loss‚0.9934560060501099 .loss/FRt2/detection_loss‚2.362568189294971 /loss/FBt1/detection_loss‚3.0393256092071534 %loss/total_loss‚67.50723799002768 _step‚110 .loss/FBt2/detection_loss‚4.323901389326368 /loss/FRt1/detection_loss‚2.8946769911309946 /loss/FRt1/assignment_loss‚5.769587402758391 0loss/FRt3/assignment_loss‚3.9472930749257404 2loss/SRqqt4/assignment_loss‚1.8550013050616505 /loss/SRqqt1/detection_loss‚3.23807498216629 epoch‚1 % _timestamp‚1.7883964759267352e+09 1loss/SRqqt1/assignment_loss‚2.409842839241028 1loss/SRqqt2/assignment_loss‚2.258121696335298 0loss/SRqqt2/detection_loss‚2.231532134391643ó°{VXÓ Ò /loss/FBt3/detection_loss‚11.936997175216675 /loss/FBt4/detection_loss‚1.3811501286029644 /loss/FRt2/assignment_loss‚4.261145883676957 /loss/FRt4/detection_loss‚2.5884445784396903 1loss/SRqqt4/detection_loss‚1.4019721318260734 0loss/FRt4/assignment_loss‚2.8866755144540654 /loss/FBt2/assignment_loss‚0.380422824861593 /loss/FRt3/detection_loss‚2.6123480664359198 0loss/SRqqt3/detection_loss‚2.090966126946516 trainer/global_step‚1349 _runtime‚1100.563873504 0loss/FBt4/assignment_loss‚0.3545848075010023 2loss/SRqqt3/assignment_loss‚1.9276423856269482 0loss/FBt1/assignment_loss‚0.3615067465496395 0loss/FBt3/assignment_loss‚0.9934560060501099 .loss/FRt2/detection_loss‚2.362568189294971 /loss/FBt1/detection_loss‚3.0393256092071534 %loss/total_loss‚67.50723799002768 _step‚110 .loss/FBt2/detection_loss‚4.323901389326368 /loss/FRt1/detection_loss‚2.8946769911309946 /loss/FRt1/assignment_loss‚5.769587402758391 0loss/FRt3/assignment_loss‚3.9472930749257404 2loss/SRqqt4/assignment_loss‚1.8550013050616505 /loss/SRqqt1/detection_loss‚3.23807498216629 epoch‚1 % _timestamp‚1.7883964759267352e+09 1loss/SRqqt1/assignment_loss‚2.409842839241028 1loss/SRqqt2/assignment_loss‚2.258121696335298 0loss/SRqqt2/detection_loss‚2.231532134391643nÏWÂ’Ô ‚(:‡ ŇãÔç†Â® gpu.0.gpu‚29.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.088' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6176 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|Ÿ^ú}Õ ‚(:ò ŇãÔ§ïÛ±" proc.memory.rssMB‚ 2676.6328125 disk.drbd2080.out‚0# disk.dm-0.out‚18411.44287109375 network.recv‚455700* disk./.usagePercent‚17.499156147965635& disk.nvme0n1.out‚18411.44287109375 disk.sda.in‚ 0.02734375 disk.drbd2265.in‚0! disk.nvme0c0n1.in‚ 334.1484375( disk.nvme0c0n1.out‚18411.44287109375 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.02734375 disk.rbd0.in‚0$ memory_percent‚88.75889778137207 network.sent‚ 1.951852e+06 disk.sda1.out‚0 disk.sda3.out‚ 2553.78515625 disk.rbd1.in‚0 disk.drbd2266.out‚0* proc.memory.percent‚16.336870193481445 cpu‚25.36163171310357 disk.nvme0n1.in‚ 334.1484375 disk.sda.out‚ 2553.78515625 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 3.75048828125 disk.drbd2080.in‚0 disk.sda3.in‚ 0.02734375 proc.cpu.threads‚17 disk.sda2.out‚0% disk./.usageGB‚260.75007247924805 disk.drbd1716.in‚0 disk.dm-1.out‚ 2553.78515625 disk.rbd0.out‚0( proc.memory.availableMB‚ 1841.7421875 disk.drbd2266.in‚0 disk.dm-0.in‚ 334.1484375§ã…HÖ ‚(:’ Ô‡ãÔî´¤® gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.841' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.568199999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ͼ½r× ‚(:ç Ô‡ãÔÔÕ¬² disk.dm-1.out‚ 2596.2265625 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.dm-0.out‚18411.80078125! disk.nvme0c0n1.in‚ 334.1484375* proc.memory.percent‚16.336870193481445% disk./.usageGB‚260.75007247924805 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda1.out‚0* disk./.usagePercent‚17.499156147965635 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.sda3.in‚ 0.02734375# disk.nvme0n1.out‚18411.80078125 network.recv‚461613 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 4.37548828125 network.sent‚ 2.060104e+06 disk.sda.out‚ 2596.2265625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.in‚ 334.1484375% disk.nvme0c0n1.out‚18411.80078125( proc.memory.availableMB‚ 1762.7421875" proc.memory.rssMB‚ 2676.6328125 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 334.1484375$ memory_percent‚89.24107551574707 cpu‚25.357755964034666 disk.sda.in‚ 0.02734375 disk.dm-1.in‚ 0.02734375 disk.sda3.out‚ 2596.2265625û‘+m2Ø - _wandbruntime‚1139  _runtime‚1139wwW ÉÙ Ã % _timestamp‚1.7883965150383403e+09 _runtime‚1139.675207347 _step‚111 ' lr-AdamW/pg1‚0.00021482491296222558 ' lr-AdamW/pg2‚0.00021482491296222558 trainer/global_step‚13999Y6ÍÚ Ç ' lr-AdamW/pg1‚0.00021482491296222558 ' lr-AdamW/pg2‚0.00021482491296222558 trainer/global_step‚1399 % _timestamp‚1.7883965150383403e+09 _runtime‚1139.675207347 _step‚111oæ²'2Û - _wandbruntime‚1139  _runtime‚1139\°–<Ü ¾O UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚41350524361728 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 1972074820608 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚43730492358144 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚18406221 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚39378280542208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 1972243819520 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚578 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8009328 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚43730728980992 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 trainer/global_step‚1399 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚39378280542208 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚41724341243904 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1972243819520 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682752 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2006151114240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚39424687959139 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28276853 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1972074820608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10745504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622848 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1966392722159 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚41350524361728 WJDeviceStatsMonitor.on_train_batch_start/allocatedBÈ_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _step‚112 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 _runtime‚1139.676627708 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28276853 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18754256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚39378262539264 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18406221 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚37458113076224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46663038 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚39378262539264 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28256817 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41724365184000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18406223 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28256817 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46683076 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18406223 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41350337359872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10745506 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1966557843555 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41350337359872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚28276853 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚18754256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚39378262539264 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚18406221 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚37458113076224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚46663038 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚39378262539264 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚28256817 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚41724365184000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚18406223 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚28256817 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚46683076 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚18406223 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚41350337359872 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚10745506 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚576 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 1966557843555 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚41350337359872 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 1972243819520 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682752 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2006151114240 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚39424687959139 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚28276853 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 1972074820608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚10745504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622848 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 1966392722159 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚41350524361728 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _step‚112 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408pFµ”Å’Þ ‚(:‡ ã‡ãÔôôž® gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚31.0 gpu.0.powerWatts‚119.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9466 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï…hß ‚(:Ý ã‡ãÔã³»± disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.drbd2265.in‚0* proc.memory.percent‚16.336870193481445! disk.nvme0c0n1.in‚ 334.1484375 network.recv‚466545 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 334.1484375 disk.rbd1.out‚ 4.84423828125$ memory_percent‚89.70677852630615 cpu‚25.353833070150305 network.sent‚ 2.066494e+06$ disk./.usageGB‚260.7500762939453 disk.dm-1.in‚ 0.02734375 disk.sda1.out‚0 disk.sda3.in‚ 0.02734375 disk.sda.in‚ 0.02734375 disk.sda2.in‚0 disk.dm-1.out‚ 2638.5859375 disk.nvme0n1.out‚ 18412.09375" proc.memory.rssMB‚ 2676.6328125 disk.rbd0.out‚0 disk.nvme0n1.in‚ 334.1484375) proc.memory.availableMB‚ 1686.44140625" disk.nvme0c0n1.out‚ 18412.09375) disk./.usagePercent‚17.49915640397317 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.sda.out‚ 2638.5859375 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.dm-0.out‚ 18412.09375 disk.sda1.in‚0 disk.sda3.out‚ 2638.5859375 disk.rbd1.in‚0ìÛ»2à - _wandbruntime‚1140  _runtime‚1140jÚØÏ&á ÉM F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1967781886703 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7883965157862349e+09 _runtime‚1140.423677931 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚41379186877952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46695484 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1973636971008 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18767626 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41753254192128 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚39405549906944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10752881 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8014743 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43760826788864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18418882 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28276604 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10752883 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46695484 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1967947008307 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2007781071872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18418880 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39405531904000 _step‚113 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 trainer/global_step‚1399 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208475136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1973467958784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18418880 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28296666 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1973636971008 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚41378999862784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39405549906944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18418882 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39451838145263 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39405531904000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18767049 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46715548 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8014168 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199u„,d„â ÍM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚28276604 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232415232 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚41378999862784 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚39405549906944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚18418882 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚39451838145263 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚39405531904000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚18767049 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚46715548 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8014168 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 1967781886703 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7883965157862349e+09 _runtime‚1140.423677931 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚41379186877952 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚46695484 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 1973636971008 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚18767626 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚41753254192128 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚39405549906944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚10752881 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8014743 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚43760826788864 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚18418882 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚28276604 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚10752883 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚46695484 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 1967947008307 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2007781071872 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚18418880 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚39405531904000 _step‚113 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 trainer/global_step‚1399 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208475136 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 1973467958784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚18418880 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚28296666 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 1973636971008 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29269555 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2041915142144 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48288690 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45247335569408 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1449 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19039171 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚553 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19039173 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚42783663921152 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48308728 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19039173 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29269555 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 _runtime‚1179.324725781 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8281573 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43169988655104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚42783476919296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8281022 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚40791136733591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚38755289232384 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚42783663921152 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚40741730776064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2035864540567 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚40741748779008 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚40741730776064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11114356 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2077322974208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚116 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2041746143232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230331392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚38755272193024 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2036029661963 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29249519 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42783476919296 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2041915142144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7883965546875665e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2077116582912 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚40791318894347 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚29269555 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2041915142144 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚48288690 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚45247335569408 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚1449 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19039171 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚553 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚19039173 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚42783663921152 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚48308728 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19039173 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚29269555 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 _runtime‚1179.324725781 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8281573 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚43169988655104 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚42783476919296 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8281022 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚40791136733591 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚38755289232384 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚42783663921152 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚40741730776064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2035864540567 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚40741748779008 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚40741730776064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11114356 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2077322974208 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚116 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2041746143232 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230331392 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚38755272193024 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2036029661963 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚29249519 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚42783476919296 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040tPEº2ð - _wandbruntime‚1180  _runtime‚1180cò•ñ ÉM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2043139281408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚40769018143744 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8286959 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚40769000140800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _step‚117 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/req¨Mz: uested_bytes.large_pool.allocated‚38781232414720 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42812326437376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2043308293632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚38781215375360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45277671994880 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29289368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2043308293632 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48341200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚1180.068643008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29269306 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2078746451456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210572288 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚40769000140800 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42812139422208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 trainer/global_step‚1449 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19051832 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 % _timestamp‚1.7883965554311619e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40818651241435 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8286442 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234512384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19408173 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11121733 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19051830 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚40769018143744 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42812326437376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45277437482496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2078535879168 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚40818469080471 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29289368Âr¶Ó&ò ÍM WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚45277437482496 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2078535879168 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚40818469080471 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚29289368 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2043139281408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚40769018143744 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8286959 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚40769000140800 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _step‚117 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚38781232414720 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚42812326437376 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2043308293632 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚38781215375360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚45277671994880 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚29289368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2043308293632 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚48341200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _runtime‚1180.068643008 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚29269306 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2078746451456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210572288 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚40769000140800 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚42812139422208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 trainer/global_step‚1449 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19051832 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 % _timestamp‚1.7883965554311619e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚40818651241435 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8286442 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234512384 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚19408173 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11121733 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19051830 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚40769018143744 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚42812326437376 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062u$2#P2ó - _wandbruntime‚1180  _runtime‚1180‹ç†Sô Í 0loss/SRqqt3/detection_loss‚2.061125200079835 2loss/SRqqt1/assignment_loss‚2.0763999345125974 trainer/global_step‚1449 _step‚118 /loss/FRt1/assignment_loss‚4.916085751433122 0loss/FRt3/assignment_loss‚3.2993914630677965 1loss/SRqqt2/assignment_loss‚1.958387003495143 /loss/FRt4/detection_loss‚2.1095560250271643 0loss/SRqqt1/detection_loss‚2.201875524940314 %loss/total_loss‚55.62954178274095 epoch‚1 _runtime‚ 1180.06984177 /loss/FBt4/detection_loss‚0.8831488896408629 2loss/SRqqt4/assignment_loss‚1.9130318236269364 /loss/FBt4/assignment_loss‚0.235644150224794 .loss/FRt1/detection_loss‚2.449459337090191 1loss/SRqqt4/detection_loss‚1.6325908089949661 0loss/FRt2/assignment_loss‚3.6486199276787894 2loss/SRqqt3/assignment_loss‚2.0983811856611916 0loss/FBt3/assignment_loss‚0.4038022307013307 /loss/FRt3/detection_loss‚2.5372772223419613 % _timestamp‚1.7883965554330442e+09 /loss/FRt4/assignment_loss‚2.672405477081026 .loss/FRt2/detection_loss‚2.901850083896092 0loss/FBt2/assignment_loss‚0.5106144249199133 /loss/FBt1/detection_loss‚3.6053206138312817 .loss/FBt2/detection_loss‚3.638920835653941 .loss/FBt3/detection_loss‚5.063030481338501 /loss/FBt1/assignment_loss‚0.344495131958638 1loss/SRqqt2/detection_loss‚2.4681282555445647ÓÎM¹Wõ Ñ 2loss/SRqqt1/assignment_loss‚2.0763999345125974 trainer/global_step‚1449 _step‚118 /loss/FRt1/assignment_loss‚4.916085751433122 0loss/FRt3/assignment_loss‚3.2993914630677965 1loss/SRqqt2/assignment_loss‚1.958387003495143 /loss/FRt4/detection_loss‚2.1095560250271643 0loss/SRqqt1/detection_loss‚2.201875524940314 %loss/total_loss‚55.62954178274095 epoch‚1 _runtime‚ 1180.06984177 /loss/FBt4/detection_loss‚0.8831488896408629 2loss/SRqqt4/assignment_loss‚1.9130318236269364 /loss/FBt4/assignment_loss‚0.235644150224794 .loss/FRt1/detection_loss‚2.449459337090191 1loss/SRqqt4/detection_loss‚1.6325908089949661 0loss/FRt2/assignment_loss‚3.6486199276787894 2loss/SRqqt3/assignment_loss‚2.0983811856611916 0loss/FBt3/assignment_loss‚0.4038022307013307 /loss/FRt3/detection_loss‚2.5372772223419613 % _timestamp‚1.7883965554330442e+09 /loss/FRt4/assignment_loss‚2.672405477081026 .loss/FRt2/detection_loss‚2.901850083896092 0loss/FBt2/assignment_loss‚0.5106144249199133 /loss/FBt1/detection_loss‚3.6053206138312817 .loss/FBt2/detection_loss‚3.638920835653941 .loss/FBt3/detection_loss‚5.063030481338501 /loss/FBt1/assignment_loss‚0.344495131958638 1loss/SRqqt2/detection_loss‚2.4681282555445647 0loss/SRqqt3/detection_loss‚2.061125200079835vx=¿Q’ö ‚(:‡ ˆãÔáÛ’® gpu.0.gpu‚34.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚31.0 gpu.0.powerWatts‚103.919' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.7838 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JØ`xy÷ ‚(:î ˆãÔËö©± disk.drbd2265.out‚0 disk.rbd0.in‚0' disk.nvme0c0n1.out‚18412.9892578125 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 2759.08203125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda3.out‚ 2759.08203125* disk./.usagePercent‚17.499202997344405 disk.sda2.out‚0 cpu‚25.3425384695581 disk.sda.in‚ 0.02734375 disk.drbd2080.out‚0! proc.memory.rssMB‚ 2676.640625 disk.sda1.in‚0 network.sent‚ 2.266539e+06 network.recv‚481784% disk./.usageGB‚260.75077056884766 disk.drbd2080.in‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 1667.05078125 disk.sda.out‚ 2759.08203125 disk.sda3.in‚ 0.02734375! disk.nvme0c0n1.in‚ 334.1484375% disk.nvme0n1.out‚18412.9892578125 disk.drbd2265.in‚0 disk.dm-0.in‚ 334.1484375" disk.dm-0.out‚18412.9892578125 disk.rbd1.in‚0 disk.nvme0n1.in‚ 334.1484375 proc.cpu.threads‚17 disk.drbd1716.out‚0$ memory_percent‚89.82512950897217* proc.memory.percent‚16.336917877197266 disk.sda1.out‚0 disk.rbd1.out‚ 5.71923828125 disk.dm-1.in‚ 0.02734375$àÿõ’ø ‚(:‡ ŸˆãԸ̥® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚101.784' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.3568 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‰6uù ‚(:ê ŸˆãÔÓ¿©±% disk./.usageGB‚260.75074768066406 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.dm-0.in‚ 334.1484375& disk.nvme0n1.out‚18413.39501953125! proc.memory.rssMB‚ 2676.640625 cpu‚25.33895650632313 disk.drbd2080.out‚0 proc.cpu.threads‚17 disk.drbd2265.out‚0# disk.dm-0.out‚18413.39501953125 disk.sda3.in‚ 0.02734375( disk.nvme0c0n1.out‚18413.39501953125 disk.sda.in‚ 0.02734375 disk.sda2.out‚0 disk.dm-1.in‚ 0.02734375 disk.dm-1.out‚ 2798.203125 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 334.1484375# memory_percent‚89.8282527923584 network.recv‚488486* disk./.usagePercent‚17.499201461299197 disk.sda.out‚ 2798.203125 disk.rbd0.in‚0 disk.sda3.out‚ 2798.203125* proc.memory.percent‚16.336917877197266 disk.drbd1716.in‚0 disk.rbd1.out‚ 5.71923828125 network.sent‚ 2.375248e+06 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 334.1484375 disk.sda2.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 1666.5390625ŒEM’ú ‚(:‡ ®ˆãÔ؉›® gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9466 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ¿Ðoû ‚(:ä ®ˆãÔçÞô± disk.drbd2265.out‚0 disk.drbd2080.in‚0* proc.memory.percent‚16.336917877197266$ disk./.usageGB‚260.7506294250488 disk.rbd1.out‚ 5.75146484375& disk.nvme0n1.out‚18414.13720703125 network.recv‚492152 disk.sda3.in‚0.03125 disk.nvme0n1.in‚ 334.1484375 proc.cpu.threads‚17 disk.dm-1.in‚0.03125 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 334.1484375 network.sent‚ 2.380597e+06 disk.sda1.out‚0 disk.dm-1.out‚ 2837.0703125 disk.drbd2265.in‚0# disk.dm-0.out‚18414.13720703125! proc.memory.rssMB‚ 2676.640625 cpu‚25.335637419838193 disk.sda3.out‚ 2837.0703125 disk.sda.out‚ 2837.0703125 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0# memory_percent‚89.8252010345459( proc.memory.availableMB‚ 1667.0390625 disk.sda1.in‚0 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 334.1484375( disk.nvme0c0n1.out‚18414.13720703125 disk.sda.in‚0.03125 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0* disk./.usagePercent‚17.499193525065635)é:2ü - _wandbruntime‚1219  _runtime‚1219þ'FÈý  ' lr-AdamW/pg2‚0.00021461003488998487 trainer/global_step‚1499 % _timestamp‚1.7883965945704558e+09 _runtime‚ 1219.20732237 _step‚119 ' lr-AdamW/pg1‚0.00021461003488998487/jüÌþ Æ trainer/global_step‚1499 % _timestamp‚1.7883965945704558e+09 _runtime‚ 1219.20732237 _step‚119 ' lr-AdamW/pg1‚0.00021461003488998487 ' lr-AdamW/pg2‚0.00021461003488998487w]žƒp2ÿ - _wandbruntime‚1219  _runtime‚1219…„Âjœ€¿O L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚30242221 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚30242221 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19672121 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8552373 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚565 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42105199012864 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46763660395008 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚567 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚42105217015808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚44216616478720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2111417465856 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20035012 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11483206 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚44615660006400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216877056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2105501480371 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚44216803480576 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚40052448349184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _step‚120 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30262257 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42105199012864 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44216616478720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11483204 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49914342 IˆÉ'ÃO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2111586464768 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8551808 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 % _timestamp‚1.7883965945711854e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚42157949829555 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚19672123 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2105336358975 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2111586464768 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚19672121 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚42157767668799 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚42105217015808 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚30262257 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚44615636066304 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚44216616478720 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11483204 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚49914342 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚19672121 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8552373 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚565 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚42105199012864 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚46763660395008 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚567 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚42105217015808 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚44216616478720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2111417465856 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20035012 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11483206 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚44615660006400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216877056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2105501480371 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚44216803480576 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚40052448349184 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _step‚120 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚30262257 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚42105199012864 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 I6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 trainer/global_step‚1499 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49966852 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49966852 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚44245465996800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2149657292800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2106890645123 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚42132486380544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2112979616256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2112979616256 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2106725523519 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42132486380544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20047744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7883965953182945e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214766592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238706688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚30282070 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11490581 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30262008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11490583 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30282070 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2112810604032 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19684780 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42132468377600 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚42185100015679 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8557697 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8557163 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚121 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚44644572954624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚44245278981632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19684780 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46793991540736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚44245465996800 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚40078391531520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 _runtime‚ 1219.95591968 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19684782 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚44644549014528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49946788 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2112810604032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2149442526208 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2íxV)ë"„ÌM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8557697 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8557163 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚121 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚44644572954624 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚44245278981632 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚19684780 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚46793991540736 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚44245465996800 ODeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚40078391531520 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 _runtime‚ 1219.95591968 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚19684782 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚44644549014528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚49946788 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2112810604032 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2149442526208 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 trainer/global_step‚1499 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚49966852 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚49966852 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚44245465996800 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2149657292800 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2106890645123 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚42132486380544 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2112979616256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2112979616256 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2106725523519 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚42132486380544 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20047744 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 262255616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7883965953182945e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214766592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238706688 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.Q1"çallocated‚30282070 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11490581 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚30262008 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚11490583 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚30282070 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2112810604032 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚19684780 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚42132468377600 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚42185100015679 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416y±‡ª/2…- _wandbruntime‚1219  _runtime‚1219„¬S†Í /loss/FRt4/assignment_loss‚3.652025832579686 _runtime‚1219.957647611 1loss/FBt4/assignment_loss‚0.23872673373080033 .loss/FRt1/detection_loss‚2.112921118736267 .loss/FRt2/detection_loss‚2.563839535628046 /loss/FRt2/assignment_loss‚4.280286976269314 2loss/SRqqt2/assignment_loss‚1.9986599426011782 2loss/SRqqt4/assignment_loss‚1.6821631793181102 0loss/FBt1/assignment_loss‚0.5775602220433379 0loss/FBt3/assignment_loss‚0.7098338007926941 .loss/FRt3/detection_loss‚2.823538362128394 .loss/FRt4/detection_loss‚2.151415737489095 .loss/FBt2/detection_loss‚3.830538237094879 trainer/global_step‚1499 /loss/FBt3/detection_loss‚3.0406291007995607 /loss/FBt4/detection_loss‚1.0518854516908387 2loss/SRqqt3/assignment_loss‚1.7919080439954995 _step‚122 % _timestamp‚1.7883965953208244e+09 /loss/FRt1/assignment_loss‚4.737907330195109 /loss/FRt3/assignment_loss‚4.453587831769671 1loss/SRqqt1/assignment_loss‚2.020959606766701 0loss/SRqqt1/detection_loss‚2.551214506228765 1loss/SRqqt4/detection_loss‚1.4076467183729013 1loss/SRqqt2/detection_loss‚2.4635608276805363 0loss/SRqqt3/detection_loss‚2.040237072482705 .loss/FBt1/detection_loss‚4.045628272570097 %loss/total_loss‚56.70788854877684 0loss/FBt2/assignment_loss‚0.4812141078126539 epoch‚1Ï êSW‡Ñ /loss/FRt1/assignment_loss‚4.737907330195109 /loss/FRt3/assignment_loss‚4.453587831769671 1loss/SRqqt1/assignment_loss‚2.020959606766701 0loss/SRqqt1/detection_loss‚2.551214506228765 1loss/SRqqt4/detection_loss‚1.4076467183729013 1loss/SRqqt2/detection_loss‚2.4635608276805363 0loss/SRqqt3/detection_loss‚2.040237072482705 .loss/FBt1/detection_loss‚4.045628272570097 %loss/total_loss‚56.70788854877684 0loss/FBt2/assignment_loss‚0.4812141078126539 epoch‚1 /loss/FRt4/assignment_loss‚3.652025832579686 _runtime‚1219.957647611 1loss/FBt4/assignment_loss‚0.23872673373080033 .loss/FRt1/detection_loss‚2.112921118736267 .loss/FRt2/detection_loss‚2.563839535628046 /loss/FRt2/assignment_loss‚4.280286976269314 2loss/SRqqt2/assignment_loss‚1.9986599426011782 2loss/SRqqt4/assignment_loss‚1.6821631793181102 0loss/FBt1/assignment_loss‚0.5775602220433379 0loss/FBt3/assignment_loss‚0.7098338007926941 .loss/FRt3/detection_loss‚2.823538362128394 .loss/FRt4/detection_loss‚2.151415737489095 .loss/FBt2/detection_loss‚3.830538237094879 trainer/global_step‚1499 /loss/FBt3/detection_loss‚3.0406291007995607 /loss/FBt4/detection_loss‚1.0518854516908387 2loss/SRqqt3/assignment_loss‚1.7919080439954995 _step‚122 % _timestamp‚1.7883965953208244e+09zEĪܒˆ‚(:‡ ½ˆãÔüž§® gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.917' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3834 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0uq Nq‰‚(:æ ½ˆãÔ‘¸± network.sent‚ 2.386661e+06 disk.sda3.out‚ 2876.62109375% disk.nvme0n1.out‚18414.4970703125 disk.sda.out‚ 2876.62109375 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚18414.4970703125 disk.drbd2266.out‚0* disk./.usagePercent‚17.499192757043033% disk./.usageGB‚260.75061798095703 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.rbd1.out‚ 5.84521484375 network.recv‚496651 disk.rbd0.in‚0 disk.dm-0.in‚ 334.1484375" disk.dm-0.out‚18414.4970703125 disk.sda3.in‚0.03125! disk.nvme0c0n1.in‚ 334.1484375 disk.dm-1.out‚ 2876.62109375! proc.memory.rssMB‚ 2676.640625 disk.sda.in‚0.03125 disk.drbd1716.out‚0 disk.dm-1.in‚0.03125 disk.drbd2266.in‚0) proc.memory.availableMB‚ 1666.96484375* proc.memory.percent‚16.336917877197266 cpu‚25.33204390611584 disk.drbd2265.out‚0$ memory_percent‚89.82565402984619 proc.cpu.threads‚17 disk.nvme0n1.in‚ 334.1484375 disk.sda2.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.drbd2265.in‚0«‚‡Š‚(:’ ̈ãÔËÄ® gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚101.666' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.333199999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘®'Ôr‹‚(:ç ̈ãÔ›þë°* disk./.usagePercent‚17.499192757043033 disk.sda.in‚0.03125 disk.sda.out‚ 2918.1171875 disk.dm-1.out‚ 2918.1171875 disk.drbd2080.out‚0$ memory_percent‚89.82751369476318) proc.memory.availableMB‚ 1666.66015625 network.recv‚502459 disk.nvme0n1.in‚ 334.1484375 disk.drbd2266.in‚0 cpu‚25.328737298112127 disk.sda1.in‚0% disk./.usageGB‚260.75061798095703 disk.dm-1.in‚0.03125 disk.drbd2080.in‚0# disk.dm-0.out‚18415.27587890625 disk.sda3.in‚0.03125! disk.nvme0c0n1.in‚ 334.1484375* proc.memory.percent‚16.336917877197266 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 5.90771484375 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.dm-0.in‚ 334.1484375 disk.rbd1.in‚0! proc.memory.rssMB‚ 2676.640625 network.sent‚ 2.494913e+06& disk.nvme0n1.out‚18415.27587890625 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚18415.27587890625 disk.sda1.out‚0 disk.sda3.out‚ 2918.1171875Óàe2Œ- _wandbruntime‚1258  _runtime‚1258G…°-È ' lr-AdamW/pg1‚0.00021446909904612241 ' lr-AdamW/pg2‚0.00021446909904612241 trainer/global_step‚1549 % _timestamp‚1.7883966342345748e+09 _runtime‚ 1258.87149873 _step‚123Œ«¼ÌŽÆ trainer/global_step‚1549 % _timestamp‚1.7883966342345748e+09 _runtime‚ 1258.87149873 _step‚123 ' lr-AdamW/pg1‚0.00021446909904612241 ' lr-AdamW/pg2‚0.00021446909904612241{§xy/2- _wandbruntime‚1258  _runtime‚1258H3kÅ'¿O UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚45649943040000 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚43524398604007 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚43468685252608 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46061307417600 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚46061283477504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20305071 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚43468685252608 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8825065 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20305073 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7883966342354085e+09 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2181088788480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚124 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20676583 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚48280527248384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11852056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1549 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20305071 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚43468667249664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31234923 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11852054 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚43524580764763 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41349590426624 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45649756038144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚1258.872536429 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2174808177383 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚45649756038144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31254959 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218974208 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31234923 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8824529 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2219000856576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚45649943040000 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48280284334080 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2181257787392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2181088788480 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚43468667249664 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31254959 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2181257787392 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20677121 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2219219830784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242914304 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2174973298779 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚20305073 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚51560032 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚41349607465984 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚43468685252608 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚46061307417600 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚46061283477504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚20305071 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚43468685252608 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚8825065 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚20305073 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7883966342354085e+09 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2181088788480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚124 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚20676583 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚48280527248384 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚11852056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚1549 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚20305071 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚43468667249664 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚31234923 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚11852054 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚43524580764763 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚41349590426624 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚45649756038144 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚1258.872536429 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2174808177383 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚45649756038144 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚31254959 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218974208 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚31234923 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚8824529 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2219000856576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚45649943040000 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚48280284334080 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2181257787392 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2181088788480 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚43468667249664 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚31254959 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2181257787392 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚20677121 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2219219830784 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242914304 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 262255616 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2174973298779|ˆ[êÛ2’- _wandbruntime‚1259  _runtime‚1259n¬µÌ“ÈM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11859431 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚46090220365824 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8829883 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51592504 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚125 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41375550648320 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8830404 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20317732 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚43495936614400 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚1549 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚46090196425728 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚43551913111851 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 S@DeviceStatsMo‡â]Ënitor.on_train_batch_end/active_bytes.all.allocated‚45678605556224 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2182650938880 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20689314 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788396634987284e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚43495954617344 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1259.624943906 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20317730 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31254710 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20317730 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2176362463531 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚45678605556224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2182481926656 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚45678418541056 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48310849321472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚43495936614400,UO-Ò&”ÌM E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚125 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚41375550648320 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚8830404 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20317732 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚43495936614400 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚1549 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚46090196425728 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚43551913111851 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚45678605556224 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2182650938880 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚20689314 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788396634987284e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚43495954617344 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚1259.624943906 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20317730 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚31254710 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20317730 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2176362463531 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚45678605556224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2182481926656 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚45678418541056 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚48310849321472 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚43495936614400 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚11859431 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚46090220365824 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚8829883 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚51592504 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217}sš€Á2•- _wandbruntime‚1259  _runtime‚1259¢@ž²U–Ï .loss/FRt2/detection_loss‚2.891117671557835 .loss/FRt1/assignment_loss‚4.21393660042021 0loss/FRt4/assignment_loss‚2.5982778569062552 /loss/FBt3/assignment_loss‚0.673585663549602 0loss/FBt4/assignment_loss‚0.3195168648453067 /loss/FRt3/detection_loss‚2.8495999460711197 1loss/SRqqt2/detection_loss‚2.6797921552851394 /loss/FBt4/detection_loss‚0.8770804466897687 % _timestamp‚1.7883966349897475e+09 1loss/FBt2/assignment_loss‚0.22998757328787178 .loss/FRt1/detection_loss‚2.337024637946376 /loss/FRt4/detection_loss‚1.9856618030203714 1loss/SRqqt1/detection_loss‚2.4121264761144463 0loss/SRqqt3/detection_loss‚2.042924556805163 2loss/SRqqt2/assignment_loss‚1.9350299255267993 /loss/FBt1/detection_loss‚3.1493559496270285 2loss/SRqqt1/assignment_loss‚1.9541887021200224 1loss/FBt1/assignment_loss‚0.49040270274761294 .loss/FBt2/detection_loss‚4.800875276327133 &loss/total_loss‚55.723520205691074 epoch‚1 _step‚126 1loss/SRqqt3/assignment_loss‚1.674021385458051 1loss/SRqqt4/detection_loss‚1.3574907465837895 trainer/global_step‚1549 /loss/FRt2/assignment_loss‚4.546169860022409 /loss/FRt3/assignment_loss‚3.532435038510491 -loss/FBt3/detection_loss‚4.67891389131546 _runtime‚1259.626537304 1loss/SRqqt4/assignment_loss‚1.494004474952817šZAÔY—Ó -loss/FBt3/detection_loss‚4.67891389131546 _runtime‚1259.626537304 1loss/SRqqt4/assignment_loss‚1.494004474952817 .loss/FRt2/detection_loss‚2.891117671557835 .loss/FRt1/assignment_loss‚4.21393660042021 0loss/FRt4/assignment_loss‚2.5982778569062552 /loss/FBt3/assignment_loss‚0.673585663549602 0loss/FBt4/assignment_loss‚0.3195168648453067 /loss/FRt3/detection_loss‚2.8495999460711197 1loss/SRqqt2/detection_loss‚2.6797921552851394 /loss/FBt4/detection_loss‚0.8770804466897687 % _timestamp‚1.7883966349897475e+09 1loss/FBt2/assignment_loss‚0.22998757328787178 .loss/FRt1/detection_loss‚2.337024637946376 /loss/FRt4/detection_loss‚1.9856618030203714 1loss/SRqqt1/detection_loss‚2.4121264761144463 0loss/SRqqt3/detection_loss‚2.042924556805163 2loss/SRqqt2/assignment_loss‚1.9350299255267993 /loss/FBt1/detection_loss‚3.1493559496270285 2loss/SRqqt1/assignment_loss‚1.9541887021200224 1loss/FBt1/assignment_loss‚0.49040270274761294 .loss/FBt2/detection_loss‚4.800875276327133 &loss/total_loss‚55.723520205691074 epoch‚1 _step‚126 1loss/SRqqt3/assignment_loss‚1.674021385458051 1loss/SRqqt4/detection_loss‚1.3574907465837895 trainer/global_step‚1549 /loss/FRt2/assignment_loss‚4.546169860022409 /loss/FRt3/assignment_loss‚3.532435038510491~5¹.^˜‚(:’ ÛˆãԪ鞮 gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚103.799' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.759800000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏO©ds™‚(:è ÛˆãÔ“ÆÔ° disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0$ memory_percent‚89.83092308044434 disk.drbd2266.in‚0 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 334.1484375 proc.cpu.threads‚17% disk./.usageGB‚260.75061798095703 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 6.26708984375 disk.sda.in‚0.03125 network.sent‚ 2.501211e+06 network.recv‚506887 disk.dm-1.in‚0.03125 disk.drbd2080.out‚0 disk.nvme0n1.in‚ 334.1484375 disk.sda.out‚ 2965.96484375 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0* proc.memory.percent‚16.336917877197266 disk.drbd2265.in‚0 disk.dm-0.in‚ 334.1484375# disk.dm-0.out‚18415.54150390625 disk.sda3.out‚ 2965.96484375( disk.nvme0c0n1.out‚18415.54150390625 disk.sda2.out‚0 disk.dm-1.out‚ 2965.96484375( proc.memory.availableMB‚ 1666.1015625 disk.sda1.in‚0 disk.sda3.in‚0.03125! proc.memory.rssMB‚ 2676.640625 cpu‚25.32568557567759* disk./.usagePercent‚17.499192757043033 disk.drbd2080.in‚0& disk.nvme0n1.out‚18415.54150390625Ý™eŽš‚(:„ êˆãÔäá®® gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚31.0 gpu.0.powerWatts‚104.45' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.89 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿ“P`›‚(:Õ êˆãÔîÕÚ° disk.sda3.out‚ 3017.703125$ disk.nvme0c0n1.out‚ 18415.8984375" disk.nvme0n1.out‚ 18415.8984375 cpu‚25.322694785677257 disk.drbd2266.out‚0 network.recv‚511983 disk.sda.in‚0.03125 disk.dm-1.out‚ 3017.703125 disk.rbd1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚0.03125 disk.sda1.in‚0* disk./.usagePercent‚17.499191477005365 disk.sda2.in‚0! proc.memory.rssMB‚ 2676.640625 disk.sda1.out‚0 disk.rbd1.out‚ 6.34521484375' proc.memory.availableMB‚ 1665.796875 disk.drbd2266.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 334.1484375 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 334.1484375! disk.nvme0c0n1.in‚ 334.1484375* proc.memory.percent‚16.336917877197266 disk.sda3.in‚0.03125$ disk./.usageGB‚260.7505989074707 disk.sda.out‚ 3017.703125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.dm-0.out‚ 18415.8984375$ memory_percent‚89.83278274536133 network.sent‚ 2.608699e+06 disk.drbd2080.out‚0]Ž»’œ‚(:‡ ùˆãÔˆþ¸® gpu.0.gpu‚29.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.552' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5104 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ©xh‚(:Ý ùˆãÔÑòȱ disk.dm-0.in‚ 334.1484375 disk.dm-1.in‚ 0.03515625) proc.memory.availableMB‚ 1666.51171875 disk.drbd2265.in‚0 disk.rbd1.out‚ 6.34521484375 disk.sda.out‚ 3067.296875 disk.sda2.in‚0! proc.memory.rssMB‚ 2676.640625 disk.rbd0.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 334.1484375! disk.nvme0n1.out‚ 18416.109375$ memory_percent‚89.82841968536377 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda3.in‚ 0.03515625 disk.sda2.out‚0 disk.drbd1716.in‚0* proc.memory.percent‚16.336917877197266* disk./.usagePercent‚17.499191733012896 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 3067.296875 disk.sda.in‚ 0.03515625 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 334.1484375# disk.nvme0c0n1.out‚ 18416.109375 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.dm-0.out‚ 18416.109375 disk.drbd1716.out‚0 disk.dm-1.out‚ 3067.296875 network.sent‚ 2.614667e+06 network.recv‚516576% disk./.usageGB‚260.75060272216797 cpu‚25.31954957965292ØÂ@±2ž- _wandbruntime‚1299  _runtime‚1299ôZu ÉŸÃ ' lr-AdamW/pg1‚0.00021430586960749913 ' lr-AdamW/pg2‚0.00021430586960749913 trainer/global_step‚1599 % _timestamp‚1.7883966743863378e+09 _runtime‚1299.023312909 _step‚127@62Í Ç ' lr-AdamW/pg1‚0.00021430586960749913 ' lr-AdamW/pg2‚0.00021430586960749913 trainer/global_step‚1599 % _timestamp‚1.7883966743863378e+09 _runtime‚1299.023312909 _step‚127û |x2¡- _wandbruntime‚1299  _runtime‚12998bªè &¢¿O E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2244279995791 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9098591 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚44891211699971 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚44832135486464 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚47082895597568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚541 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚44832153489408 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216877056 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12220904 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32247661 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53185684 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240817152 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 _runtime‚1299.024763537 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53165646 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚47082895597568 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚541 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚44832153489408 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216877056 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12220904 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚32247661 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚53185684 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240817152 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 _runtime‚1299.024763537 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚53165646 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21332572 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20950680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53218156 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53198092 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44918361886095 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚42672709765120 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚49827394006016 % _timestamp‚1.7883966751264844e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2252153249280 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32267474 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32267474 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2291526228992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚44859422854144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20950682 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚47111745115648 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2245834281939 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2245669160335 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2252153249280 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212669440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20950682 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53198092 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20950680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47535843836928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32247412 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2252322261504 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚47111558100480 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚129 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21332035 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 trainer/global_step‚1599%×<Ô&¦ÎM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12228283 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚47111745115648 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚49827157396480 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2252322261504 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚47111558100480 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚129 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21332035 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 trainer/global_step‚1599 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚53218156 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2252322261504 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚47535867777024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21332572 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚20950680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚53218156 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚53198092 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚44918361886095 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚42672709765120 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚49827394006016 % _timestamp‚1.7883966751264844e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2252153249280 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚32267474 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚32267474 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2291526228992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚44859422854144 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚20950682 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚47111745115648 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2245834281939 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2245669160335 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2252153249280 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212669440 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚20950682 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚53198092 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚20950680 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚47535843836928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚32247412 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48516035156992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 _step‚132 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21962680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33220327 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21570973 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21570973 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚46195621726208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33240363 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚51313628698112 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2320600432640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚553 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 % _timestamp‚1.7883967141830344e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12589756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12589754 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9372373 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚48516222158848 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2320600432640 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚51313398366720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚46195603723264 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230331392 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚48952578299904 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚46257842635179 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2313751814199 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206391296 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43943908660224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚551 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚46195621726208 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54791298 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21962127 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21570971 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33240363 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54811336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2313916935595 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _runtime‚1338.820148611 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33220327 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚48952602240000Ã0Ê'µÄO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚21570973 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚46195621726208 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚33240363 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚51313628698112 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2320600432640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚553 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 % _timestamp‚1.7883967141830344e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚12589756 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12589754 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9372373 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚48516222158848 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2320600432640 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚51313398366720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚46195603723264 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230331392 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚48952578299904 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚46257842635179 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2313751814199 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206391296 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚43943908660224 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚551 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚46195621726208 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚54791298 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚21962127 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚21570971 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚33240363 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚54811336 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2313916935595 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _runtime‚1338.820148611 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚33220327 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚48952602240000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚43943925699584 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚48516222158848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2361026458112 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚48516035156992 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚46257660474423 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9372924 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚48516035156992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 _step‚132 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚21962680 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚33220327 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚21570973 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248„ ÿš×2¶- _wandbruntime‚1339  _runtime‚1339~¤yM·ÉM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚33260176 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚54823744 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21583632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54843808 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚46222891090944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2321993584128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48544697659904 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚51343966336000 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚48981515188224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 % _timestamp‚1.7883967149341571e+09 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54843808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43969851842560 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚48544884675072 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚43969868881920 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21975382 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21974846 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2362240575488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33260176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21583632 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9377715 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 XFDeviceStatsMoni¸–´‚ tor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2315140978743 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12597131 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21583630 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9378249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚46222873088000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210572288 _step‚133 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2362451147776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2321993584128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚48981491248128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚21583632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚54843808 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚46222891090944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2321993584128 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚48544697659904 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚51343966336000 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚48981515188224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 % _timestamp‚1.7883967149341571e+09 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚54843808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚43969851842560 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚48544884675072 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚43969868881920 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚21975382 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚21974846 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2362240575488 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚33260176 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚21583632 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9377715 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2315140978743 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12597131 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚21583630 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9378249 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚46222873088000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210572288 _step‚133 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2362451147776 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2321993584128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚48981491248128 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49949174716416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2390102756352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34233065 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚52829918009856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50398225711104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9645825 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2383388754003 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34233065 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45241084816384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2383223632607 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47624473570387 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚49949361718272 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚549 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚136°?Žâ‰ ÅÃO Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚49949174716416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2390102756352 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚34233065 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚52829918009856 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚50398225711104 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 SGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚9645825 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2383388754003 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚34233065 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚45241084816384 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2383223632607 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚47624473570387 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚49949361718272 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚549 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚136 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚56436988 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 % _timestamp‚1.7883967542602017e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚12958604 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22216580 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚49978024234496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚47586341324800 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34252878 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56469460 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47651805917475 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9651755 _step‚137 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34232816 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34252878 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9651226 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚49977837219328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22216580 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22216582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22216582 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47651623756511 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34232816 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56469460 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22617738 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22617207 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚50427138659328 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚49978024234496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12965983 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244998144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52860488190976 _runtime‚1379.653993737 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50427162599424\ÒÖÓ&ÈÍM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚47586341324800 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2391495894528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚12965981 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 221058048 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚56449396 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚52860243192832 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚45267027998720 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2391664906752 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2433104533504 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2384777918755 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚34252878 QEDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚9651226 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚49977837219328 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22216580 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚22216582 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚22216582 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚47651623756511 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚34232816 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚56469460 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚22617738 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚22617207 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚50427138659328 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚49978024234496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚12965983 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244998144 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚52860488190976 _runtime‚1379.653993737 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚50427162599424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚22216580 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚49978024234496 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚47586341324800 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚34252878 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚56469460 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚47651805917475 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 UIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚9651755 _step‚137 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚34232816 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 % _timestamp‚1.7883967945546725e+09 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13327454 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2502603446272 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚48922540196864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35205731 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚48991104505595 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚51382501277696 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2502805643264 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚46538226893824 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚48922558199808 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23246560 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 _step‚140 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2459774078976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚561 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9919104 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚48922540196864 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚48990922344839 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚48922558199808 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚54346702705664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51382314275840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2459774078976 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226137088 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚51843897062400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 trainer/global_step‚1749 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22836873 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22836871 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13327454 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2502603446272 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚48922540196864 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚35205731 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚48991104505595 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚51382501277696 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2502805643264 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚46538226893824 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚48922558199808 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23246560 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 _step‚140 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2459774078976 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚561 WKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚9919104 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚48922540196864 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚48990922344839 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚48922558199808 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚54346702705664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚51382314275840 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2459774078976 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226137088 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚51843897062400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 trainer/global_step‚1749 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚22836873 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚22836871 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 % _timestamp‚1.7883967945546725e+09ŒKòI°2Ø- _wandbruntime‚1419  _runtime‚1419q  ©‹%ÙÉM ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2504016379904 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23259507 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚22849530 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2461336229376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2461167217152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23258963 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58095112 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35225518 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58095112 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35225518 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚49018436852683 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2461336229376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚51410976778752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2454249737163 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚49018254691719 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2454084615559 trainer/global_step‚1749 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚51411163793920 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58075048 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚54377041157120 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚51411163793920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚48949827564544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 % _timestamp‚1.7883967953132477e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚542 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35245580 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚48949809561600 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214766592 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238706688 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚54376802450432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13334833 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13334831 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚48949827564544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2504231146496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35245580 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚46564187115520 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _step‚141 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58075048 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2461336229376 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2461167217152 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23258963 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚58095112 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚35225518 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚58095112 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚35225518 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚544 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚49018436852683 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2461336229376 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚51410976778752 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2454249737163 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚49018254691719 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2454084615559 trainer/global_step‚1749 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚51411163793920 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚58075048 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚54377041157120 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚51411163793920 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚48949827564544 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 % _timestamp‚1.7883967953132477e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚542 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚35245580 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚48949809561600 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214766592 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238706688 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚54376802450432 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13334833 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13334831 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚48949827564544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2504231146496 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚35245580 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚46564187115520 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _step‚141 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚58075048 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚23259507 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0?…Ðú2Û- _wandbruntime‚1419  _runtime‚1419dúúŸVÜÐ 0loss/FRt2/assignment_loss‚3.8382897236767937 2loss/SRqqt1/assignment_loss‚2.0661188221496083 2loss/SRqqt2/assignment_loss‚2.1296193175845675 2loss/SRqqt4/assignment_loss‚1.4985542118549346 /loss/FRt1/detection_loss‚2.3172393602629504 1loss/SRqqt2/detection_loss‚2.2802978581852384 epoch‚1 _runtime‚1419.951814486 $ _timestamp‚1.788396795315011e+09 0loss/FRt4/assignment_loss‚1.8968352270126343 0loss/SRqqt1/detection_loss‚2.113914621912915 /loss/FBt3/detection_loss‚4.0040231347084045 &loss/total_loss‚53.528019463576584 trainer/global_step‚1749 /loss/FRt4/detection_loss‚2.1623055759072303 /loss/FBt4/detection_loss‚0.7605611826008294 0loss/FRt3/assignment_loss‚3.9105247321228185 0loss/FBt2/assignment_loss‚0.7423528648085065 /loss/FBt3/assignment_loss‚1.089506141775928 .loss/FRt2/detection_loss‚2.746041346998776 2loss/SRqqt3/assignment_loss‚2.1233866700659627 0loss/SRqqt3/detection_loss‚2.042283262895501 0loss/FRt1/assignment_loss‚4.8094850306709604 .loss/FRt3/detection_loss‚2.177111388494571 0loss/FBt1/assignment_loss‚0.6258010443505975 /loss/FBt4/assignment_loss‚0.267442412983251 -loss/FBt1/detection_loss‚2.65342299754803 1loss/SRqqt4/detection_loss‚1.3060565689578651 .loss/FBt2/detection_loss‚3.966845966047711 _step‚142G÷k¥[ÝÕ 2loss/SRqqt2/assignment_loss‚2.1296193175845675 2loss/SRqqt4/assignment_loss‚1.4985542118549346 /loss/FRt1/detection_loss‚2.3172393602629504 1loss/SRqqt2/detection_loss‚2.2802978581852384 epoch‚1 _runtime‚1419.951814486 $ _timestamp‚1.788396795315011e+09 0loss/FRt4/assignment_loss‚1.8968352270126343 0loss/SRqqt1/detection_loss‚2.113914621912915 /loss/FBt3/detection_loss‚4.0040231347084045 &loss/total_loss‚53.528019463576584 trainer/global_step‚1749 /loss/FRt4/detection_loss‚2.1623055759072303 /loss/FBt4/detection_loss‚0.7605611826008294 0loss/FRt3/assignment_loss‚3.9105247321228185 0loss/FBt2/assignment_loss‚0.7423528648085065 /loss/FBt3/assignment_loss‚1.089506141775928 .loss/FRt2/detection_loss‚2.746041346998776 2loss/SRqqt3/assignment_loss‚2.1233866700659627 0loss/SRqqt3/detection_loss‚2.042283262895501 0loss/FRt1/assignment_loss‚4.8094850306709604 .loss/FRt3/detection_loss‚2.177111388494571 0loss/FBt1/assignment_loss‚0.6258010443505975 /loss/FBt4/assignment_loss‚0.267442412983251 -loss/FBt1/detection_loss‚2.65342299754803 1loss/SRqqt4/detection_loss‚1.3060565689578651 .loss/FBt2/detection_loss‚3.966845966047711 _step‚142 0loss/FRt2/assignment_loss‚3.8382897236767937 2loss/SRqqt1/assignment_loss‚2.0661188221496083Ž&\ö’Þ‚(:‡ €ŠãÔ«ò•® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚112.997' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5994 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0M\Arrß‚(:ç €ŠãÔ˜ˆ¬± disk.nvme0n1.in‚ 334.1484375 disk.sda.in‚ 0.0390625 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.0390625 disk.dm-1.out‚ 3418.06640625 cpu‚25.29497865905393 network.recv‚558817 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 6.65185546875% proc.memory.availableMB‚ 1666.0625* proc.memory.percent‚16.336917877197266% disk.nvme0n1.out‚18419.5849609375 disk.sda.out‚ 3418.06640625) disk./.usagePercent‚17.49916280416153 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0$ memory_percent‚89.83116149902344" disk.dm-0.out‚18419.5849609375 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 334.1484375 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0! proc.memory.rssMB‚ 2676.640625 network.sent‚ 2.973817e+06! disk.nvme0c0n1.in‚ 334.1484375 disk.sda3.out‚ 3418.06640625 disk.sda1.out‚0 disk.sda3.in‚ 0.0390625' disk.nvme0c0n1.out‚18419.5849609375 disk.drbd2265.in‚0% disk./.usageGB‚260.75017166137695 proc.cpu.threads‚17Þ”/’à‚(:‡ ŠãÔ—²”® gpu.0.gpu‚30.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.907' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7814 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®f¶Øwá‚(:ì ŠãÔ‡Ñþ± network.recv‚565633 disk.drbd2080.in‚0 disk.dm-0.in‚ 334.1484375' disk.nvme0c0n1.out‚18419.9169921875 disk.dm-1.out‚ 3492.13671875 disk.drbd2266.out‚0$ memory_percent‚89.83156681060791* proc.memory.percent‚16.336917877197266" disk.dm-0.out‚18419.9169921875 disk.sda1.out‚0 disk.nvme0n1.in‚ 334.1484375 disk.sda.out‚ 3492.13671875 disk.sda2.in‚0 disk.sda2.out‚0% disk./.usageGB‚260.75017166137695 disk.drbd2080.out‚0 disk.sda3.in‚ 0.0390625 disk.rbd1.out‚ 6.73779296875% disk.nvme0n1.out‚18419.9169921875 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.0390625 disk.drbd2265.in‚0) disk./.usagePercent‚17.49916280416153 disk.rbd0.out‚0 disk.drbd1716.out‚0) proc.memory.availableMB‚ 1665.99609375 disk.sda3.out‚ 3492.13671875 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 334.1484375 disk.sda.in‚ 0.0390625 disk.drbd2265.out‚0 proc.cpu.threads‚17 network.sent‚ 3.082597e+06 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd2266.in‚0! proc.memory.rssMB‚ 2676.640625 cpu‚25.292446848106003”-É-’â‚(:‡ žŠãÔ–í§® gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.088' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6176 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0/®_‘mã‚(:â žŠãÔª‡Ö° network.recv‚569152) disk./.usagePercent‚17.49915461192043# disk.nvme0n1.out‚18420.36328125 disk.sda.out‚ 3544.28125 disk.drbd1716.in‚0 disk.drbd2266.out‚0* proc.memory.percent‚16.336941719055176 disk.dm-0.out‚18420.36328125 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.dm-0.in‚ 334.1484375 cpu‚25.290036161146684 proc.cpu.threads‚17 disk.dm-1.out‚ 3544.28125 disk.drbd2080.out‚0 disk.rbd1.out‚ 6.80029296875% disk.nvme0c0n1.out‚18420.36328125 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 334.1484375$ memory_percent‚89.83500003814697 disk.sda1.out‚0 disk.sda3.in‚ 0.04296875 disk.sda3.out‚ 3544.28125 disk.nvme0n1.in‚ 334.1484375 disk.sda.in‚ 0.04296875 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.04296875) proc.memory.availableMB‚ 1665.43359375 network.sent‚ 3.087934e+06% disk./.usageGB‚260.75004959106445 disk.drbd2080.in‚0# proc.memory.rssMB‚ 2676.64453125 disk.rbd0.in‚0 disk.rbd0.out‚0£ŠyE2ä- _wandbruntime‚1459  _runtime‚1459É2ñ€Èå _runtime‚1459.263682026 _step‚143 ' lr-AdamW/pg1‚0.00021343076577444588 ' lr-AdamW/pg2‚0.00021343076577444588 trainer/global_step‚1799 $ _timestamp‚1.788396834626786e+09+tÔ ÍæÇ _runtime‚1459.263682026 _step‚143 ' lr-AdamW/pg1‚0.00021343076577444588 ' lr-AdamW/pg2‚0.00021343076577444588 trainer/global_step‚1799 $ _timestamp‚1.788396834626786e+09×ýà2ç- _wandbruntime‚1459  _runtime‚1459Ë¡üÇ'èÁO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚59668254 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚50286026436608 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚50357553280047 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚47835386010624 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚23469823 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 trainer/global_step‚1799 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚50286008433664 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622848 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2573771516416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚36218469 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚53289544473600 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚50357735440803 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚59688292 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682752 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13696304 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13696306 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10191165 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23469821 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚52815640837120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2529614400512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2529614400512 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55863315990016 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55863079367168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚47835403049984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23888002 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10191696 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2529445401600 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52815453835264 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2522332390819 _runtime‚1459.264758328 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2522167269423 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚52815640837120 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23469821 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36198433 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7883968346275902e+09 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682752 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚13696304 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚13696306 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10191165 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚23469821 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚52815640837120 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2529614400512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2529614400512 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚55863315990016 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚55863079367168 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚47835403049984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚23888002 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10191696 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2529445401600 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚52815453835264 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2522332390819 _runtime‚1459.264758328 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2522167269423 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚52815640837120 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚23469821 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚36198433 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 % _timestamp‚1.7883968346275902e+09 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚52844303353344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212669440 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚50384885626927 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59700700 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10197183 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚47861346232320 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36238282 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23482480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13703681 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚53318433481728 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚50385067787891 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23900329 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2523721555571 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚52844303353344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13703683 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59700700 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚50313295801344 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36218220 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 _runtime‚1460.020784918 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚47861329192960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7883968353833177e+09 _step‚145 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2530838539776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55893408912384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚53318457421824 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2530838539776 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59720764 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚50313277798400 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55893645521920 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2574975430656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23482482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚50313295801344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217.­Â…Ö&ìÐM MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚55893408912384 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚53318457421824 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2530838539776 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚59720764 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚50313277798400 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚55893645521920 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2574975430656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚23482482 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚50313295801344 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚52844303353344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212669440 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚50384885626927 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚59700700 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10197183 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚47861346232320 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚36238282 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚23482480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚13703681 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚53318433481728 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚50385067787891 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚23900329 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2523721555571 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚52844303353344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚13703683 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚59700700 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚50313295801344 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚36218220 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 _runtime‚1460.020784918 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚47861329192960 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7883968353833177e+09 _step‚145 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2530838539776 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040‘ÞL•º2í- _wandbruntime‚1460  _runtime‚1460?Û[0\îÖ epoch‚1 /loss/FRt2/detection_loss‚2.7333303139065253 /loss/FRt4/assignment_loss‚1.932603029709942 1loss/FBt1/assignment_loss‚0.47183651197016896 .loss/FBt2/detection_loss‚3.992086927096049 &loss/total_loss‚53.564579983165366 1loss/FBt2/assignment_loss‚0.49305004286725307 % _timestamp‚1.7883968353852084e+09 /loss/FRt1/assignment_loss‚4.470123306298867 /loss/FBt1/detection_loss‚2.8753779144848095 /loss/FBt3/detection_loss‚2.8064909956671973 trainer/global_step‚1799 0loss/FRt3/assignment_loss‚3.8548386096954346 1loss/FBt4/assignment_loss‚0.21305193307123477 2loss/SRqqt1/assignment_loss‚2.0794610524055908 2loss/SRqqt4/assignment_loss‚1.5379887245762258 0loss/FBt3/assignment_loss‚0.3955905322324146 /loss/FRt1/detection_loss‚2.7190608206467752 1loss/SRqqt3/detection_loss‚2.3367594044382978 1loss/SRqqt2/assignment_loss‚2.336131432721781 /loss/FRt3/detection_loss‚2.5108505537112555 0loss/SRqqt1/detection_loss‚2.101281607029389 1loss/SRqqt2/detection_loss‚2.4315037727355957 /loss/FBt4/detection_loss‚1.1840622288593636 _runtime‚1460.022026081 _step‚146 /loss/FRt2/assignment_loss‚4.339042161450242 .loss/FRt4/detection_loss‚2.028673644616919 2loss/SRqqt3/assignment_loss‚2.2278501994726136 1loss/SRqqt4/detection_loss‚1.4935342635014164duV&aïÛ _runtime‚1460.022026081 _step‚146 /loss/FRt2/assignment_loss‚4.339042161450242 .loss/FRt4/detection_loss‚2.028673644616919 2loss/SRqqt3/assignment_loss‚2.2278501994726136 1loss/SRqqt4/detection_loss‚1.4935342635014164 epoch‚1 /loss/FRt2/detection_loss‚2.7333303139065253 /loss/FRt4/assignment_loss‚1.932603029709942 1loss/FBt1/assignment_loss‚0.47183651197016896 .loss/FBt2/detection_loss‚3.992086927096049 &loss/total_loss‚53.564579983165366 1loss/FBt2/assignment_loss‚0.49305004286725307 % _timestamp‚1.7883968353852084e+09 /loss/FRt1/assignment_loss‚4.470123306298867 /loss/FBt1/detection_loss‚2.8753779144848095 /loss/FBt3/detection_loss‚2.8064909956671973 trainer/global_step‚1799 0loss/FRt3/assignment_loss‚3.8548386096954346 1loss/FBt4/assignment_loss‚0.21305193307123477 2loss/SRqqt1/assignment_loss‚2.0794610524055908 2loss/SRqqt4/assignment_loss‚1.5379887245762258 0loss/FBt3/assignment_loss‚0.3955905322324146 /loss/FRt1/detection_loss‚2.7190608206467752 1loss/SRqqt3/detection_loss‚2.3367594044382978 1loss/SRqqt2/assignment_loss‚2.336131432721781 /loss/FRt3/detection_loss‚2.5108505537112555 0loss/SRqqt1/detection_loss‚2.101281607029389 1loss/SRqqt2/detection_loss‚2.4315037727355957 /loss/FBt4/detection_loss‚1.1840622288593636’õ:Õƒ’ð‚(:‡ ­ŠãÔžµµ® gpu.0.gpu‚19.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚101.907' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.3814 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0*%Írñ‚(:ç ­ŠãÔ”Ѭ±# memory_percent‚89.8352861404419 cpu‚25.287771137008644 disk.drbd1716.in‚0" disk.nvme0n1.out‚ 18420.6171875) proc.memory.availableMB‚ 1665.38671875 disk.drbd2266.out‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.nvme0n1.in‚ 334.1484375 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.sda3.in‚ 0.04296875 disk.rbd1.in‚0# proc.memory.rssMB‚ 2676.64453125 network.sent‚ 3.093956e+06 disk.sda.in‚ 0.04296875 disk.dm-1.out‚ 3595.05859375 disk.dm-0.out‚ 18420.6171875 disk.sda1.out‚0 disk.rbd1.out‚ 6.80029296875* proc.memory.percent‚16.336941719055176) disk./.usagePercent‚17.49916280416153 disk.drbd2265.in‚0 disk.dm-0.in‚ 334.1484375 disk.sda.out‚ 3595.05859375 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 334.1484375$ disk.nvme0c0n1.out‚ 18420.6171875% disk./.usageGB‚260.75017166137695 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0 network.recv‚573584 disk.dm-1.in‚ 0.04296875 disk.rbd0.in‚0 disk.sda3.out‚ 3595.05859375“Umˆ’ò‚(:‡ ¼ŠãԽϦ® gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2658 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ùÃiyó‚(:î ¼ŠãÔÿà ± disk.sda3.in‚ 0.04296875 disk.rbd1.out‚ 6.83251953125 disk.sda2.out‚0 disk.dm-1.in‚ 0.04296875 disk.dm-1.out‚ 3647.046875 disk.sda1.out‚0 disk.sda3.out‚ 3647.046875 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0# proc.memory.rssMB‚ 2676.64453125& disk.nvme0n1.out‚18421.01904296875 disk.sda.out‚ 3647.046875 disk.rbd0.out‚0 cpu‚25.285578263971146 network.recv‚579830! disk.nvme0c0n1.in‚ 334.1484375 disk.sda.in‚ 0.04296875 network.sent‚ 3.202125e+06# disk.dm-0.out‚18421.01904296875 disk.drbd2266.out‚0 disk.drbd2080.in‚0$ memory_percent‚89.83380794525146) proc.memory.availableMB‚ 1665.62890625 disk.rbd1.in‚0( disk.nvme0c0n1.out‚18421.01904296875 disk.nvme0n1.in‚ 334.1484375 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0% disk./.usageGB‚260.75017166137695* proc.memory.percent‚16.336941719055176 proc.cpu.threads‚17) disk./.usagePercent‚17.49916280416153 disk.dm-0.in‚ 334.1484375 disk.sda1.in‚0xF2ô- _wandbruntime‚1499  _runtime‚1499¼™ïŽÈõ trainer/global_step‚1849 $ _timestamp‚1.788396874741742e+09 _runtime‚1499.378648392 _step‚147 ' lr-AdamW/pg1‚0.00021315667126420271 ' lr-AdamW/pg2‚0.00021315667126420271[ô®ÍöÇ $ _timestamp‚1.788396874741742e+09 _runtime‚1499.378648392 _step‚147 ' lr-AdamW/pg1‚0.00021315667126420271 ' lr-AdamW/pg2‚0.00021315667126420271 trainer/global_step‚1849“kcá 2÷- _wandbruntime‚1499  _runtime‚1499‡93Ó¶øÁO @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚49132562166784 _runtime‚1499.379646266 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚24102771 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 HBDeviceSquO&tatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2599116724224 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54248593394688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24102773 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37211171 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚57379595604480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51649494673408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54735167944704 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230331392 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24102771 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2599285723136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10466691 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2644634051072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24531309 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14065156 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 % _timestamp‚1.7883968747425137e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚51724184215255 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚10466155 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206391296 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚51649494673408 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2591639087831 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚54248593394688 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24102773 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚37211171 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚57379595604480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚51649494673408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚54735167944704 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230331392 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24102771 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2599285723136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10466691 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2644634051072 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚24531309 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14065156 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0”P§Š#’ú‚(:‡ ËŠãÔú—Œ® gpu.0.gpu‚29.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.088' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6176 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0A70º}û‚(:ò ËŠãÔϞΰ network.sent‚ 3.208394e+06 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚18421.61572265625& disk.nvme0n1.out‚18421.61572265625 disk.sda.in‚ 0.04296875 network.recv‚584695 disk.drbd2265.in‚0 disk.sda.out‚ 3695.99609375# proc.memory.rssMB‚ 2676.64453125 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 6.83251953125! disk.nvme0c0n1.in‚ 334.1484375# memory_percent‚89.8353099822998 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.04296875 disk.dm-1.out‚ 3695.99609375 disk.dm-0.in‚ 334.1484375# disk.dm-0.out‚18421.61572265625 disk.sda1.out‚0 disk.nvme0n1.in‚ 334.1484375* proc.memory.percent‚16.336941719055176) disk./.usagePercent‚17.49916280416153 disk.rbd1.in‚0% disk./.usageGB‚260.75017166137695 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚ 0.04296875 disk.sda3.out‚ 3695.99609375( proc.memory.availableMB‚ 1665.3828125 cpu‚25.283119001816555ÿ2ü- _wandbruntime‚1500  _runtime‚1500 ë#GÑ&ýËM _step‚149 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37210922 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51676764038144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 247095296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223155200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24115430 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚49158505349120 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61326352 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚54277442912768 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚54277255897600 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚548 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2600509862400 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚57410162424832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37230984 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚54277442912768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2593193373979 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2600678874624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10472129 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37230984 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51751698723099 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _runtime‚1500.126521039 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51676746035200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2600509862400 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24115432 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚54764080892928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61346416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2646057591808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24544662 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚51676764038144 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24544112 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14072531 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37210922 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54764104833024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51751516562135 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0l }“àþÐM O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24115430 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2593028252375 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2600678874624 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚24544112 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14072531 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚37210922 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚54764104833024 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚51751516562135 SEDeviceStatsMonitor.on_train_batch_end/requestedtÂL2ö!_bytes.small_pool.peak‚ 406789961 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _step‚149 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚37210922 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚51676764038144 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 247095296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223155200 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚24115430 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚49158505349120 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚61326352 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚54277442912768 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚54277255897600 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚548 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2600509862400 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚57410162424832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚37230984 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚54277442912768 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2593193373979 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2600678874624 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚10472129 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚37230984 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚51751698723099 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _runtime‚1500.126521039 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚51676746035200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2600509862400 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚24115432 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚54764080892928 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚61346416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2646057591808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚24544662 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚51676764038144 F^Ø 2loss/SRqqt1/assignment_loss‚2.1161522827035673 0loss/FBt3/assignment_loss‚0.6258085817094525 .loss/FBt1/detection_loss‚3.064714350483634 0loss/SRqqt4/detection_loss‚1.471191035117954 1loss/FBt4/assignment_loss‚0.24969453086888524 /loss/FRt4/detection_loss‚2.2552303104173568 .loss/FBt2/detection_loss‚4.010842305421829 epoch‚1 /loss/FRt4/assignment_loss‚2.701197258063725 1loss/SRqqt1/detection_loss‚2.2338399500460238 1loss/SRqqt2/detection_loss‚2.0723695913139655 trainer/global_step‚1849 /loss/FBt1/assignment_loss‚0.336435501259323 0loss/SRqqt3/detection_loss‚1.851612347960472 /loss/FBt3/detection_loss‚3.2082967281341555 _step‚150 2loss/SRqqt2/assignment_loss‚2.4478961557758097 _runtime‚1500.127639831 /loss/FRt1/assignment_loss‚3.826849937438965 /loss/FRt3/assignment_loss‚4.167707745621844 2loss/SRqqt4/assignment_loss‚1.5330313194426708 1loss/FBt2/assignment_loss‚0.19324821082045673 /loss/FRt1/detection_loss‚2.3418905984711005 /loss/FRt3/detection_loss‚2.6663804737533012 /loss/FBt4/detection_loss‚1.2034741041600294 % _timestamp‚1.7883968754908192e+09 /loss/FRt2/assignment_loss‚4.119861320751469 2loss/SRqqt3/assignment_loss‚1.9226393723487853 /loss/FRt2/detection_loss‚2.6119453601720855 &loss/total_loss‚53.232309372256864–~xê’‚‚(:‡ ÚŠãÔ棘® gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚105.684' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.1368 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 1Ýtƒ‚(:é ÚŠãÔ€­ª± disk.drbd2080.in‚0( disk.nvme0c0n1.out‚18422.36083984375# proc.memory.rssMB‚ 2676.64453125 proc.cpu.threads‚17% disk./.usageGB‚260.75017166137695 disk.drbd1716.out‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 334.1484375 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.sda3.in‚0.046875! disk.nvme0c0n1.in‚ 334.1484375 cpu‚25.280986006320163 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda.out‚ 3739.97265625 disk.sda2.in‚0 disk.dm-1.out‚ 3739.97265625 disk.drbd2266.in‚0 disk.nvme0n1.in‚ 334.1484375$ memory_percent‚89.83421325683594 disk.sda.in‚0.046875 disk.dm-1.in‚0.046875 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 6.8955078125& disk.nvme0n1.out‚18422.36083984375* proc.memory.percent‚16.336941719055176# disk.dm-0.out‚18422.36083984375 disk.sda3.out‚ 3739.97265625% proc.memory.availableMB‚ 1665.5625 network.sent‚ 3.304695e+06 disk.drbd2080.out‚0 network.recv‚589935) disk./.usagePercent‚17.49916280416153 disk.drbd2266.out‚0 æà’„‚(:‡ éŠãÔóʱ® gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.022' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4044 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ðv»x…‚(:í éŠãÔªšÌ²# proc.memory.rssMB‚ 2676.64453125* disk./.usagePercent‚17.499163060169064 disk.sda.out‚ 3782.21484375 disk.sda2.in‚0 disk.drbd2265.out‚0$ memory_percent‚89.83423709869385 network.sent‚ 3.402346e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0$ disk./.usageGB‚260.7501754760742 disk.sda2.out‚0 disk.drbd2266.in‚0 cpu‚25.278876110449488 proc.cpu.threads‚17 disk.dm-1.in‚0.046875 disk.drbd2080.out‚0* proc.memory.percent‚16.336941719055176 network.recv‚595798 disk.drbd1716.in‚0 disk.dm-1.out‚ 3782.21484375! disk.nvme0c0n1.in‚ 334.1484375) proc.memory.availableMB‚ 1665.55859375 disk.sda3.out‚ 3782.21484375 disk.sda.in‚0.046875 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.dm-0.in‚ 334.1484375# disk.dm-0.out‚18422.57568359375( disk.nvme0c0n1.out‚18422.57568359375 disk.drbd2265.in‚0 disk.sda1.out‚0 disk.sda3.in‚0.046875 disk.rbd1.out‚ 7.2080078125 disk.nvme0n1.in‚ 334.1484375 disk.rbd1.in‚0& disk.nvme0n1.out‚18422.57568359375#Wæî2†- _wandbruntime‚1539  _runtime‚1539ÝßÏíƇÀ _step‚151 & lr-AdamW/pg1‚0.0002128605566671814 & lr-AdamW/pg2‚0.0002128605566671814 trainer/global_step‚1899 % _timestamp‚1.7883969148912768e+09 _runtime‚ 1539.52810995mN\ßˈŠ& lr-AdamW/pg2‚0.0002128605566671814 trainer/global_step‚1899 % _timestamp‚1.7883969148912768e+09 _runtime‚ 1539.52810995 _step‚151 & lr-AdamW/pg1‚0.0002128605566671814—ÆûkU2‰- _wandbruntime‚1539  _runtime‚1539Ó‡Q Æ'ŠÀO ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62919558 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚56180839296000 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24735723 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2661110906239 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55681732954112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚53090815150463 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62939596 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234525696 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚1899 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚53090997311219 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55681919955968 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚53012944907264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14434006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚50429721283584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 $ _timestamp‚1.788396914892016e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55681919955968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2668957045760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2668957045760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25172586 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2715459841536 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10739102 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2661276027635 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24735721 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14434004 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38183837 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399ŸÒÞ«ô‹ÅO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234525696 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 trainer/global_step‚1899 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚53090997311219 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚55681919955968 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚53012944907264 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚14434006 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚50429721283584 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 $ _timestamp‚1.788396914892016e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚55681919955968 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2668957045760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2668957045760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚25172586 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2715459841536 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚152 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚10739102 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2661276027635 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚24735721 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚14434004 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚38183837 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚62919558 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚56180839296000 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.currSŽ‰-× ent‚ 165121396 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚24735723 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2661110906239 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚55681732954112 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚53090815150463 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚62939596 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0˜•'-2Œ- _wandbruntime‚1540  _runtime‚1540?t€Ñ&ËM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚56209728304128 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62972068 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2670350197248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2716668124160 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62972068 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 % _timestamp‚1.7883969156403224e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58926396428288 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2662665192387 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53040232274944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53040214272000 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38223686 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _step‚153 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53118329658307 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2662500070783 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2670181185024 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10744138 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50455647426560 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2670350197248 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2716880793600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚55710395457024 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50455664465920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24748380 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚541 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38223686 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14441381 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _runtime‚1540.277767866 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55710582472192 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25185519 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58926633037824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 trainer/global_step‚1899‹Õ`éÖ&ŽÐM H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2662665192387 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚53040232274944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚53040214272000 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚38223686 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _step‚153 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚53118329658307 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2662500070783 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2670181185024 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚10744138 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚50455647426560 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2670350197248 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2716880793600 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚55710395457024 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚50455664465920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚24748380 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚541 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚38223686 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚14441381 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _runtime‚1540.277767866 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚55710582472192 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25185519 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚58926633037824 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 trainer/global_step‚1899 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚56209728304128 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚62972068 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2670350197248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2716668124160 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚62972068 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 % _timestamp‚1.7883969156403224e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚58926396428288™áPŸg2- _wandbruntime‚1540  _runtime‚1540ˆà´YÓ epoch‚1 0loss/FBt1/assignment_loss‚0.7196592851118608 .loss/FRt2/detection_loss‚3.157065591641835 1loss/SRqqt1/detection_loss‚1.9654401550916107 _runtime‚1540.279344163 /loss/FRt1/assignment_loss‚4.176784430037845 /loss/FRt2/assignment_loss‚5.532001980713436 2loss/SRqqt3/assignment_loss‚1.9492337376527165 2loss/SRqqt4/assignment_loss‚1.4938620863153653 1loss/SRqqt2/detection_loss‚2.4553008794784548 trainer/global_step‚1899 2loss/SRqqt2/assignment_loss‚2.4054436862468718 0loss/FBt3/assignment_loss‚0.7302073510363698 /loss/FBt3/detection_loss‚2.9506366699934006 $ _timestamp‚1.788396915642546e+09 /loss/FRt4/assignment_loss‚2.191640579892743 /loss/FRt3/detection_loss‚2.4108001326139155 0loss/FRt3/assignment_loss‚3.8702599578178845 1loss/SRqqt4/detection_loss‚1.0353779819081812 &loss/total_loss‚53.387935794109254 .loss/FBt2/detection_loss‚3.318432309797832 /loss/FBt4/detection_loss‚0.9468799393870155 1loss/SRqqt1/assignment_loss‚1.993988292122429 0loss/FBt2/assignment_loss‚0.5729877984324828 /loss/FRt4/detection_loss‚1.7783229693530067 _step‚154 1loss/FBt4/assignment_loss‚0.30188405403304736 .loss/FRt1/detection_loss‚2.490903905846856 1loss/SRqqt3/detection_loss‚1.9808927158946577 /loss/FBt1/detection_loss‚2.9599293036894365E%%^‘Ø 2loss/SRqqt2/assignment_loss‚2.4054436862468718 0loss/FBt3/assignment_loss‚0.7302073510363698 /loss/FBt3/detection_loss‚2.9506366699934006 $ _timestamp‚1.788396915642546e+09 /loss/FRt4/assignment_loss‚2.191640579892743 /loss/FRt3/detection_loss‚2.4108001326139155 0loss/FRt3/assignment_loss‚3.8702599578178845 1loss/SRqqt4/detection_loss‚1.0353779819081812 &loss/total_loss‚53.387935794109254 .loss/FBt2/detection_loss‚3.318432309797832 /loss/FBt4/detection_loss‚0.9468799393870155 1loss/SRqqt1/assignment_loss‚1.993988292122429 0loss/FBt2/assignment_loss‚0.5729877984324828 /loss/FRt4/detection_loss‚1.7783229693530067 _step‚154 1loss/FBt4/assignment_loss‚0.30188405403304736 .loss/FRt1/detection_loss‚2.490903905846856 1loss/SRqqt3/detection_loss‚1.9808927158946577 /loss/FBt1/detection_loss‚2.9599293036894365 epoch‚1 0loss/FBt1/assignment_loss‚0.7196592851118608 .loss/FRt2/detection_loss‚3.157065591641835 1loss/SRqqt1/detection_loss‚1.9654401550916107 _runtime‚1540.279344163 /loss/FRt1/assignment_loss‚4.176784430037845 /loss/FRt2/assignment_loss‚5.532001980713436 2loss/SRqqt3/assignment_loss‚1.9492337376527165 2loss/SRqqt4/assignment_loss‚1.4938620863153653 1loss/SRqqt2/detection_loss‚2.4553008794784548 trainer/global_step‚1899š÷ç´¾’’‚(:‡ øŠãÔ·‰¨® gpu.0.gpu‚30.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.377' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6754 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ô™]w“‚(:ì øŠãÔš¼Å± disk.sda2.out‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 3815.42578125 disk.sda3.out‚ 3815.42578125 disk.nvme0n1.in‚ 334.1484375& disk.nvme0n1.out‚18422.98388671875$ memory_percent‚89.83442783355713# proc.memory.rssMB‚ 2676.64453125 disk.sda1.out‚0 disk.sda3.in‚0.046875( disk.nvme0c0n1.out‚18422.98388671875 network.recv‚600011* disk./.usagePercent‚17.499163316176595 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.sda.out‚ 3815.42578125) proc.memory.availableMB‚ 1665.52734375 proc.cpu.threads‚17 disk.drbd2080.in‚0# disk.dm-0.out‚18422.98388671875 disk.rbd1.out‚ 7.3564453125 disk.sda.in‚0.046875* proc.memory.percent‚16.336941719055176 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 334.1484375 disk.dm-0.in‚ 334.1484375 disk.dm-1.in‚0.046875 disk.drbd2265.in‚0 disk.rbd0.out‚0 cpu‚25.27656536552796 network.sent‚ 3.408376e+06$ disk./.usageGB‚260.7501792907715=y-ô”‚(:… ‡‹ãÔ¥ª™® gpu.0.gpu‚29.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.13' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.026 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]~? x•‚(:í ‡‹ãÔûÈÓ° network.sent‚ 3.517149e+06 disk.drbd2265.in‚0# disk.dm-0.out‚18423.21044921875 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 7.3564453125( disk.nvme0c0n1.out‚18423.21044921875# proc.memory.rssMB‚ 2676.64453125* proc.memory.percent‚16.336941719055176 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 334.1484375( proc.memory.availableMB‚ 1665.0234375 disk.nvme0n1.in‚ 334.1484375 disk.sda.in‚0.046875 disk.sda2.in‚0 disk.dm-1.out‚ 3874.05859375 disk.drbd2265.out‚0 disk.drbd2080.in‚0! disk.nvme0c0n1.in‚ 334.1484375 cpu‚25.274450416016947 network.recv‚606440 disk.dm-1.in‚0.046875 disk.drbd2266.out‚0 disk.sda3.out‚ 3874.05859375 disk.rbd1.in‚0$ memory_percent‚89.83750343322754 disk.sda1.in‚0 disk.sda3.in‚0.046875* disk./.usagePercent‚17.499167156289612% disk./.usageGB‚260.75023651123047 disk.sda.out‚ 3874.05859375 disk.drbd2266.in‚0& disk.nvme0n1.out‚18423.21044921875 disk.drbd1716.out‚0 proc.cpu.threads‚17N$¸Ý–‚(:’ –‹ãÔŽ¬¹° gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.311' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.462200000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ã-mós—‚(:è –‹ãÔ¥—ö² disk.rbd1.in‚0 disk.drbd2080.in‚0 disk.sda1.out‚0$ memory_percent‚89.83447551727295* proc.memory.percent‚16.336941719055176 proc.cpu.threads‚17 disk.dm-1.in‚0.046875 disk.rbd0.in‚0 disk.sda.out‚ 3905.51953125 network.sent‚ 3.52249e+06' disk.nvme0c0n1.out‚18423.5498046875 disk.sda2.in‚0 disk.dm-1.out‚ 3905.51953125 disk.dm-0.in‚ 334.1484375 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0" disk.dm-0.out‚18423.5498046875 disk.sda1.in‚0) proc.memory.availableMB‚ 1665.51953125 disk.sda3.in‚0.046875 disk.nvme0n1.in‚ 334.1484375 disk.rbd0.out‚0# proc.memory.rssMB‚ 2676.64453125 disk.rbd1.out‚ 7.388671875 cpu‚25.272427932283808 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0$ disk./.usageGB‚260.7501792907715! disk.nvme0c0n1.in‚ 334.1484375% disk.nvme0n1.out‚18423.5498046875 network.recv‚609851* disk./.usagePercent‚17.499163316176595 disk.sda3.out‚ 3905.51953125 disk.sda.in‚0.046875 disk.drbd2266.in‚0É42˜- _wandbruntime‚1579  _runtime‚1579ŠŽUsÇ™Á _runtime‚1579.824455633 _step‚155 & lr-AdamW/pg1‚0.0002125424836932201 & lr-AdamW/pg2‚0.0002125424836932201 trainer/global_step‚1949 % _timestamp‚1.7883969551875253e+09–9»ÌšÆ trainer/global_step‚1949 % _timestamp‚1.7883969551875253e+09 _runtime‚1579.824455633 _step‚155 & lr-AdamW/pg1‚0.0002125424836932201 & lr-AdamW/pg2‚0.0002125424836932201›c¾€~2›- _wandbruntime‚1579  _runtime‚1579³xNƒÑœÀO C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39196575 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64565248 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11012783 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 QIDeviceStatsMonitor.on_train_batch_star_È®õõt/inactive_split.small_pool.current‚516 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39196575 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206391296 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25815639 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2738628368384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230331392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51726880400384 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39176539 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25368671 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2786136794112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _step‚156 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2738459369472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚51726863361024 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25368673 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚57626462767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚54376431147008 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚54376413144064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57115059515392 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57114872513536 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11012267 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚54457628246427 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 1579.82554507 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚54376413144064 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚60412599561216 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2738628368384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2730582724647 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39176539 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64545210 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25368671 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57114872513536 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248^N¢¢Ë'ÅO dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚57626486707200 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11012783 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚516 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚39196575 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206391296 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚25815639 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2738628368384 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230331392 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚51726880400384 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚39176539 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚25368671 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2786136794112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _step‚156 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2738459369472 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚51726863361024 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚25368673 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚57626462767104 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚54376431147008 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚54376413144064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚57115059515392 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚57114872513536 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11012267 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚54457628246427 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚ 1579.82554507 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚54376413144064 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚60412599561216 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2738628368384 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2730582724647 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚39176539 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚64545210 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚25368671 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚57114872513536 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚39196575 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚64565248 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚57143722031616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚60442929266176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚51752823582720 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚54403682508800 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚57143535016448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚550 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚60443165875712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236609536 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚54403700511744 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2739852507648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39196326 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25828438 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚1949 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11017655 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64597720 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39196326 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚54403700511744 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2740021519872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51752806543360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64597720 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64577656 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25381332 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2787553550848 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212669440 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64577656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2740021519872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25381330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚57655375715328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2731971889191 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39216388 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11018205 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2787766220288 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25381332 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚54484960593515 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25827886 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2739852507648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2732137010795 _runtime‚1580.566799562 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚54403682508800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚54484778432551 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25381330 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚57143722031616 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚54403682508800 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚54484778432551 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚25381330 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚57143722031616 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚57143722031616 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚60442929266176 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚51752823582720 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚54403682508800 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚57143535016448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚550 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚60443165875712 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236609536 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚54403700511744 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2739852507648 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 _step‚157 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚39196326 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚25828438 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚1949 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11017655 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚64597720 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚39196326 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚54403700511744 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2740021519872 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚51752806543360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚64597720 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚64577656 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚25381332 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2787553550848 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212669440 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚64577656 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2740021519872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚25381330 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚57655375715328 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2731971889191 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚39216388 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11018205 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2787766220288 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚25381332 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚54484960593515 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚25827886 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2739852507648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0m/ú"2¡- _wandbruntime‚1580  _runtime‚1580T£N.U¢Ï 2loss/SRqqt2/assignment_loss‚2.2747846391142867 0loss/FBt2/assignment_loss‚0.4359830650958299 .loss/FRt1/detection_loss‚2.400954687214912 /loss/FRt4/detection_loss‚1.9260900893582489 .loss/FBt4/detection_loss‚0.923862970736511 trainer/global_step‚1949 0loss/SRqqt1/assignment_loss‚1.80046071154329 2loss/SRqqt4/assignment_loss‚1.7952102829630558 1loss/FBt1/assignment_loss‚0.48386981031489995 0loss/FBt4/assignment_loss‚0.1501534921711027 _step‚158 /loss/FBt1/detection_loss‚2.4435206539928913 &loss/total_loss‚51.459202404360994 /loss/FRt1/assignment_loss‚4.134692687937554 0loss/FRt4/assignment_loss‚2.5699079823943802 0loss/FBt3/assignment_loss‚0.6621587350964546 .loss/FRt2/detection_loss‚2.683492413163185 1loss/SRqqt1/detection_loss‚1.9020211016232114 1loss/SRqqt2/detection_loss‚2.3869225455493464 1loss/SRqqt3/detection_loss‚1.8268926055343062 0loss/SRqqt4/detection_loss‚1.372227151462665 epoch‚1 _runtime‚1580.567911432 2loss/SRqqt3/assignment_loss‚1.8110170033242967 .loss/FBt2/detection_loss‚4.593823501041958 -loss/FBt3/detection_loss‚2.91951576868693 0loss/FRt3/assignment_loss‚3.2020104009854164 /loss/FRt2/assignment_loss‚4.283666777610779 .loss/FRt3/detection_loss‚2.475963327445482 % _timestamp‚1.7883969559311402e+09¿I°áZ£Ô 0loss/FRt4/assignment_loss‚2.5699079823943802 0loss/FBt3/assignment_loss‚0.6621587350964546 .loss/FRt2/detection_loss‚2.683492413163185 1loss/SRqqt1/detection_loss‚1.9020211016232114 1loss/SRqqt2/detection_loss‚2.3869225455493464 1loss/SRqqt3/detection_loss‚1.8268926055343062 0loss/SRqqt4/detection_loss‚1.372227151462665 epoch‚1 _runtime‚1580.567911432 2loss/SRqqt3/assignment_loss‚1.8110170033242967 .loss/FBt2/detection_loss‚4.593823501041958 -loss/FBt3/detection_loss‚2.91951576868693 0loss/FRt3/assignment_loss‚3.2020104009854164 /loss/FRt2/assignment_loss‚4.283666777610779 .loss/FRt3/detection_loss‚2.475963327445482 % _timestamp‚1.7883969559311402e+09 2loss/SRqqt2/assignment_loss‚2.2747846391142867 0loss/FBt2/assignment_loss‚0.4359830650958299 .loss/FRt1/detection_loss‚2.400954687214912 /loss/FRt4/detection_loss‚1.9260900893582489 .loss/FBt4/detection_loss‚0.923862970736511 trainer/global_step‚1949 0loss/SRqqt1/assignment_loss‚1.80046071154329 2loss/SRqqt4/assignment_loss‚1.7952102829630558 1loss/FBt1/assignment_loss‚0.48386981031489995 0loss/FBt4/assignment_loss‚0.1501534921711027 _step‚158 /loss/FBt1/detection_loss‚2.4435206539928913 &loss/total_loss‚51.459202404360994 /loss/FRt1/assignment_loss‚4.134692687937554žÑVx¤‚(:’ ¥‹ãÔºáé° gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.203' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.840600000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09ª‚µh¥‚(:Ý ¥‹ãÔš“Ž³' disk.nvme0c0n1.out‚18423.8271484375 disk.drbd2080.out‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 3952.5625# proc.memory.rssMB‚ 2676.64453125 proc.cpu.threads‚17 network.recv‚614412$ disk./.usageGB‚260.7504234313965 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda3.in‚0.046875$ memory_percent‚89.83490467071533" disk.dm-0.out‚18423.8271484375 disk.sda1.in‚0 disk.nvme0n1.in‚ 334.1484375 disk.sda.out‚ 3952.5625 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 334.1484375% disk.nvme0n1.out‚18423.8271484375 disk.drbd2265.out‚0 disk.sda1.out‚0 disk.sda.in‚0.046875 disk.dm-0.in‚ 334.1484375* proc.memory.percent‚16.336941719055176 cpu‚25.270355671848357 disk.drbd2265.in‚0 network.sent‚ 3.528486e+06 disk.drbd1716.in‚0 disk.dm-1.in‚0.046875 disk.dm-1.out‚ 3952.5625 disk.rbd1.out‚ 7.388671875) proc.memory.availableMB‚ 1665.44921875* disk./.usagePercent‚17.499179700658786’6¹¦‚(:’ ´‹ãÔËäÄ® gpu.0.gpu‚34.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚103.799' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.759800000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àP*¾f§‚(:Û ´‹ãÔÂد±$ disk./.usageGB‚260.7504234313965 disk.dm-1.in‚0.046875 disk.drbd2265.in‚0 disk.sda1.out‚0 disk.sda3.in‚0.046875( disk.nvme0c0n1.out‚18424.06787109375 disk.drbd2080.out‚0 disk.rbd0.in‚0# disk.dm-0.out‚18424.06787109375 disk.nvme0n1.in‚ 334.1484375 disk.sda2.out‚0 disk.dm-0.in‚ 334.1484375 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda.out‚3999.25# proc.memory.rssMB‚ 2676.64453125 network.recv‚620584* disk./.usagePercent‚17.499179700658786 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.sda3.out‚3999.25 disk.rbd1.out‚ 7.45166015625 network.sent‚ 3.63672e+06 disk.drbd2266.in‚0$ memory_percent‚89.83495235443115 disk.sda2.in‚0 disk.dm-1.out‚3999.25* proc.memory.percent‚16.336941719055176 cpu‚25.268537556449154 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 334.1484375& disk.nvme0n1.out‚18424.06787109375 disk.sda.in‚0.046875) proc.memory.availableMB‚ 1665.44140625ˆñÝ2¨- _wandbruntime‚1619  _runtime‚1619_¿£¦Ç©Á & lr-AdamW/pg2‚0.0002122025186282499 trainer/global_step‚1999 % _timestamp‚1.7883969948080165e+09 _runtime‚1619.444886828 _step‚159 & lr-AdamW/pg1‚0.0002122025186282499Q›¬ÌªÆ _runtime‚1619.444886828 _step‚159 & lr-AdamW/pg1‚0.0002122025186282499 & lr-AdamW/pg2‚0.0002122025186282499 trainer/global_step‚1999 % _timestamp‚1.7883969948080165e+09Ÿümh—2«- _wandbruntime‚1619  _runtime‚1619„UÇ'¬ÁO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2808299691008 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚40189277 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚66190900 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2808130692096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚58548012072960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26001623 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26456628 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚61929187025408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40169241 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚61929423648256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15171704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 trainer/global_step‚1999 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚55739899383808 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚55824259181635 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚59072134118400 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚558 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11284924 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66170862 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40169241 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26001623 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26457186 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _step‚160 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚58548199074816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2808130692096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682752 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚53024039517184 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2800219664451 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2800054543055 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40189277 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚58548012072960 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚556 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚58548199074816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚55739899383808 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚59072110178304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2857289529856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚58548199074816 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚55739899383808 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚59072110178304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2857289529856 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚58548012072960 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚26001623 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚26456628 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚61929187025408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚40169241 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚61929423648256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15171704 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 trainer/global_step‚1999 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚55739899383808 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚55824259181635 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚59072134118400 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚558 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11284924 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚66170862 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚40169241 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26001623 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚26457186 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _step‚160 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚58548199074816 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2808130692096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682752 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚53024039517184 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2800219664451 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2800054543055 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚40189277 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚58548012072960 õéJÅh®‚(:‡ ËãÔ·­µ¯ gpu.0.gpu‚29.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.022' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4044 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrorÊVÆR*s‚0# gpu.0.uncorrectedMemoryErrors‚0w%–3y¯‚(:î ËãÔ´›ï± disk.sda3.in‚0.046875% disk./.usageGB‚260.75048065185547 disk.drbd2266.in‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 1665.1953125* proc.memory.percent‚16.336941719055176 disk.sda.out‚ 4043.37890625 disk.dm-0.in‚ 334.1484375# disk.dm-0.out‚18424.33349609375 disk.rbd1.in‚0 disk.nvme0n1.in‚ 334.1484375# proc.memory.rssMB‚ 2676.64453125 disk.sda2.in‚0& disk.nvme0n1.out‚18424.33349609375 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 7.45166015625( disk.nvme0c0n1.out‚18424.33349609375 disk.sda3.out‚ 4043.37890625* disk./.usagePercent‚17.499183540771803 disk.sda.in‚0.046875 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.in‚0.046875 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 334.1484375 cpu‚25.266528635315282 network.sent‚ 3.642096e+06 disk.dm-1.out‚ 4043.37890625 network.recv‚624514 disk.drbd2266.out‚0$ memory_percent‚89.83645439147949 proc.cpu.threads‚17›/ú52°- _wandbruntime‚1620  _runtime‚1620÷¯@èÐ&±ÊM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚58576861591040 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11290896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40209090 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚55767168748544 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66203308 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚55767150745600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11290361 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66223372 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚61959749685248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232415232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26014282 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2858702618624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚55767168748544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚55851591528723 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40189028 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚59101047066624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208475136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2801443707599 _step‚161 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40189028 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2858494143488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55851409367759 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2809523830272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2809692842496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚53049965660160 trainer/global_step‚1999 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66203308 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15179081 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚58576674575872 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788396995564509e+09 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26014280 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚55767150745600 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚58576861591040 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26469442 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15179083 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚53049982699520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66223372 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26014282 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 $ _timestamp‚1.788396995564509e+09 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26014280 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚55767150745600 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚58576861591040 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚26469442 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15179083 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚53049982699520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚66223372 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚26014282 O6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚58576861591040 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚11290896 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚40209090 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚55767168748544 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚66203308 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚55767150745600 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11290361 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚66223372 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚61959749685248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232415232 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26014282 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2858702618624 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚55767168748544 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚55851591528723 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚40189028 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚59101047066624 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208475136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2801443707599 _step‚161 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚40189028 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2858494143488 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚55851409367759 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2809523830272 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2809692842496 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚53049965660160 trainer/global_step‚1999 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚66203308 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0¡ïXc2³- _wandbruntime‚1620  _runtime‚1620á U´Ï trainer/global_step‚1999 0loss/FRt2/assignment_loss‚4.6823770821094515 2loss/SRqqt3/assignment_loss‚1.5847687852524577 0loss/FBt1/assignment_loss‚0.6179907064288273 .loss/FRt1/detection_loss‚2.212186865011851 .loss/FRt3/detection_loss‚2.522109110688054 1loss/SRqqt3/detection_loss‚2.0585340073420886 _runtime‚1620.202947955 0loss/FRt3/assignment_loss‚3.9052826360214588 .loss/FBt4/assignment_loss‚0.31732299879557 /loss/FBt1/detection_loss‚2.9055109202861784 /loss/FBt3/detection_loss‚3.4081772327423097 $ _timestamp‚1.788396995566188e+09 _step‚162 0loss/FBt2/assignment_loss‚0.4795872878599362 /loss/FRt2/detection_loss‚2.6496610179543496 /loss/FBt4/detection_loss‚0.9707718314776663 2loss/SRqqt4/assignment_loss‚1.1325329861274975 0loss/FBt3/assignment_loss‚0.6355149388284189 .loss/FRt4/detection_loss‚1.727465921640396 %loss/total_loss‚52.30314656699717 /loss/FRt1/assignment_loss‚4.033291840553284 /loss/FRt4/assignment_loss‚2.683734973271688 1loss/SRqqt2/assignment_loss‚2.136279069795841 1loss/SRqqt1/detection_loss‚2.1548124253749847 1loss/SRqqt2/detection_loss‚2.4262575797918364 1loss/SRqqt4/detection_loss‚1.3340432953966015 epoch‚1 /loss/FBt2/detection_loss‚3.8206930028067694 1loss/SRqqt1/assignment_loss‚1.904240051439653^æ5ZµÔ _runtime‚1620.202947955 0loss/FRt3/assignment_loss‚3.9052826360214588 .loss/FBt4/assignment_loss‚0.31732299879557 /loss/FBt1/detection_loss‚2.9055109202861784 /loss/FBt3/detection_loss‚3.4081772327423097 $ _timestamp‚1.788396995566188e+09 _step‚162 0loss/FBt2/assignment_loss‚0.4795872878599362 /loss/FRt2/detection_loss‚2.6496610179543496 /loss/FBt4/detection_loss‚0.9707718314776663 2loss/SRqqt4/assignment_loss‚1.1325329861274975 0loss/FBt3/assignment_loss‚0.6355149388284189 .loss/FRt4/detection_loss‚1.727465921640396 %loss/total_loss‚52.30314656699717 /loss/FRt1/assignment_loss‚4.033291840553284 /loss/FRt4/assignment_loss‚2.683734973271688 1loss/SRqqt2/assignment_loss‚2.136279069795841 1loss/SRqqt1/detection_loss‚2.1548124253749847 1loss/SRqqt2/detection_loss‚2.4262575797918364 1loss/SRqqt4/detection_loss‚1.3340432953966015 epoch‚1 /loss/FBt2/detection_loss‚3.8206930028067694 1loss/SRqqt1/assignment_loss‚1.904240051439653 trainer/global_step‚1999 0loss/FRt2/assignment_loss‚4.6823770821094515 2loss/SRqqt3/assignment_loss‚1.5847687852524577 0loss/FBt1/assignment_loss‚0.6179907064288273 .loss/FRt1/detection_loss‚2.212186865011851 .loss/FRt3/detection_loss‚2.522109110688054 1loss/SRqqt3/detection_loss‚2.0585340073420886¢.·Q‘¶‚(:† Ò‹ãÔŸ´˜® gpu.0.gpu‚19.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.625' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.325 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k¬Ždq·‚(:æ Ò‹ãÔ‘áš± disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 7.5146484375! disk.nvme0c0n1.in‚ 334.1484375 disk.sda.in‚0.046875 disk.sda.out‚ 4078.25390625& proc.memory.availableMB‚ 1665.40625 disk.nvme0n1.in‚ 334.1484375 disk.dm-1.in‚0.046875 disk.drbd2080.in‚0 disk.dm-0.in‚ 334.1484375 disk.sda1.in‚0* disk./.usagePercent‚17.499179700658786 disk.drbd1716.out‚0 disk.dm-1.out‚ 4078.25390625 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.drbd2266.out‚0 disk.sda2.out‚0$ memory_percent‚89.83516693115234 network.sent‚ 3.738348e+06 network.recv‚629530$ disk./.usageGB‚260.7504234313965 disk.drbd1716.in‚0" disk.dm-0.out‚18424.7236328125 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚0.046875% disk.nvme0n1.out‚18424.7236328125# proc.memory.rssMB‚ 2676.64453125* proc.memory.percent‚16.336941719055176 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.sda3.out‚ 4078.25390625' disk.nvme0c0n1.out‚18424.7236328125 cpu‚25.26459282208176››a¸‚(:’ á‹ãÔ«œ® gpu.0.gpu‚22.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚31.0 gpu.0.powerWatts‚101.136' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.227199999999996 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ôïr¹‚(:ç á‹ãÔòÀ²& proc.memory.availableMB‚ 1665.40625 cpu‚25.262791859617128 network.sent‚ 3.835792e+06 disk.dm-1.in‚0.046875 disk.drbd2080.out‚0 disk.sda3.in‚0.046875 disk.sda3.out‚ 4135.58203125 disk.rbd1.in‚0 disk.nvme0n1.in‚ 334.1484375 disk.sda.in‚0.046875 network.recv‚635235 disk.drbd2265.out‚0 disk.sda1.in‚0$ memory_percent‚89.83516693115234 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0" disk.dm-0.out‚18424.9189453125 disk.drbd1716.in‚0 disk.drbd1716.out‚0* proc.memory.percent‚16.336941719055176$ disk./.usageGB‚260.7504234313965 disk.dm-0.in‚ 334.1484375 disk.rbd1.out‚ 7.5146484375 disk.sda2.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 4135.58203125% disk.nvme0n1.out‚18424.9189453125 disk.sda2.in‚0# proc.memory.rssMB‚ 2676.64453125* disk./.usagePercent‚17.499179700658786 disk.drbd2266.in‚0 disk.rbd0.out‚0' disk.nvme0c0n1.out‚18424.9189453125 disk.dm-1.out‚ 4135.58203125 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 334.1484375-_ö2º- _wandbruntime‚1659  _runtime‚1659a†‰fɻà ' lr-AdamW/pg1‚0.00021184073232048074 ' lr-AdamW/pg2‚0.00021184073232048074 trainer/global_step‚2049 % _timestamp‚1.7883970348349862e+09 _runtime‚1659.471866628 _step‚163¸,òuÎ¼È ' lr-AdamW/pg1‚0.00021184073232048074 ' lr-AdamW/pg2‚0.00021184073232048074 trainer/global_step‚2049 % _timestamp‚1.7883970348349862e+09 _runtime‚1659.471866628 _step‚163£Öžpì2½- _wandbruntime‚1659  _runtime‚1659äï+ß"¾ÀO N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 _runtime‚1659.472928061 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚57103367620608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26634571 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41161943 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚57190707956087 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚164 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2927983226368 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2869526361463 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚57103349617664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27098984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41181979 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚63445977438720 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚54321181594624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2877802014720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚63445740815872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59981338634240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚2049 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27099557 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 @8DeviceS,B ë¤tatsMonitor.on_train_batch_start/num_device_alloc‚416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2928195909120 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚57103367620608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15540554 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60517757589504 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682752 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41161943 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59981151632384 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26634573 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2877971013632 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67816552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11558430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26634571 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2877802014720 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67796514 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11559001 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚571 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59981338634240 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚57190890116843 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚573 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15540556.WOË'¿ÅO E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚67796514 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11559001 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚571 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚59981338634240 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚57190890116843 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚573 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15540556 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 _runtime‚1659.472928061 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚57103367620608 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚26634571 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚41161943 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚57190707956087 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚164 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2927983226368 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2869526361463 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚57103349617664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚27098984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚41181979 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚63445977438720 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚54321181594624 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2877802014720 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚63445740815872 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚59981338634240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚2049 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27099557 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 263586816 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 2928195909120 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚57103367620608 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15540554 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚60517757589504 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682752 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622848 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚41161943 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚59981151632384 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚26634573 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2877971013632 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚67816552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11558430 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚26634571 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2877802014720¤Õ÷²>2À- _wandbruntime‚1660  _runtime‚1660(ôµoÑ&ÁËM FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41181730 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _step‚165 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚63476307965952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26647230 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41201792 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27112104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11564173 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚60546670537728 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26647232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41181730 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2870915526007 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2879364165120 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67849024 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67828960 _runtime‚1660.236563552 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚57130618982400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚60546694477824 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚57218040302967 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67828960 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚54347124776960 trainer/global_step‚2049 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2929405012992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27112636 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232415232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2879364165120 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚54347141816320 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚60010001150464 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚57130636985344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚57218222463931 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41201792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2929613488128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚57130636985344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15547933 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67849024 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208475136RI¬kÂÐM >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚57218040302967 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚67828960 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚54347124776960 trainer/global_step‚2049 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 2929405012992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27112636 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232415232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 2879364165120 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚54347141816320 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚60010001150464 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚57130636985344 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚57218222463931 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚41201792 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 2929613488128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚57130636985344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚15547933 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚67849024 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 263586816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208475136 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚41181730 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _step‚165 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚63476307965952 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚26647230 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚41201792 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27112104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚11564173 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚60546670537728 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚26647232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚41181730 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDevic½©k[k eStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2870915526007 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 2879364165120 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚67849024 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚67828960 _runtime‚1660.236563552 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚57130618982400 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚60546694477824 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408¥¡€+t2Ã- _wandbruntime‚1660  _runtime‚1660Î]ýSÄÍ .loss/FRt2/detection_loss‚2.950954821801955 .loss/FRt3/detection_loss‚2.444185842573643 2loss/SRqqt1/assignment_loss‚2.0638219482368894 0loss/SRqqt1/detection_loss‚2.249264645576477 %loss/total_loss‚52.32928565013374 /loss/FRt1/assignment_loss‚4.833138055271572 /loss/FRt2/assignment_loss‚4.355393425110848 1loss/SRqqt4/detection_loss‚1.3015975337475538 .loss/FBt1/detection_loss‚2.410779431462288 _step‚166 2loss/SRqqt4/assignment_loss‚1.4090463746753004 .loss/SRqqt2/detection_loss‚2.1855291167895 epoch‚1 .loss/FRt4/assignment_loss‚2.02750108562983 0loss/FBt1/assignment_loss‚0.6406098247857049 0loss/FBt2/assignment_loss‚0.5066462345184846 /loss/FRt4/detection_loss‚1.8856073927420836 /loss/FBt2/detection_loss‚3.5131566060913935 /loss/FBt3/detection_loss‚2.7154663630894254 % _timestamp‚1.7883970356010408e+09 2loss/SRqqt2/assignment_loss‚2.4886138359705607 1loss/FBt4/assignment_loss‚0.24537916780857477 /loss/FRt1/detection_loss‚2.5905565809872417 1loss/SRqqt3/detection_loss‚1.9405840975046158 trainer/global_step‚2049 /loss/FRt3/assignment_loss‚3.694911667704582 1loss/SRqqt3/assignment_loss‚2.060000587999821 0loss/FBt3/assignment_loss‚0.8187683788793427 /loss/FBt4/detection_loss‚0.9977726311760616 _runtime‚1660.237863055ŠˆËOXÅÒ 2loss/SRqqt1/assignment_loss‚2.0638219482368894 0loss/SRqqt1/detection_loss‚2.249264645576477 %loss/total_loss‚52.32928565013374 /loss/FRt1/assignment_loss‚4.833138055271572 /loss/FRt2/assignment_loss‚4.355393425110848 1loss/SRqqt4/detection_loss‚1.3015975337475538 .loss/FBt1/detection_loss‚2.410779431462288 _step‚166 2loss/SRqqt4/assignment_loss‚1.4090463746753004 .loss/SRqqt2/detection_loss‚2.1855291167895 epoch‚1 .loss/FRt4/assignment_loss‚2.02750108562983 0loss/FBt1/assignment_loss‚0.6406098247857049 0loss/FBt2/assignment_loss‚0.5066462345184846 /loss/FRt4/detection_loss‚1.8856073927420836 /loss/FBt2/detection_loss‚3.5131566060913935 /loss/FBt3/detection_loss‚2.7154663630894254 % _timestamp‚1.7883970356010408e+09 2loss/SRqqt2/assignment_loss‚2.4886138359705607 1loss/FBt4/assignment_loss‚0.24537916780857477 /loss/FRt1/detection_loss‚2.5905565809872417 1loss/SRqqt3/detection_loss‚1.9405840975046158 trainer/global_step‚2049 /loss/FRt3/assignment_loss‚3.694911667704582 1loss/SRqqt3/assignment_loss‚2.060000587999821 0loss/FBt3/assignment_loss‚0.8187683788793427 /loss/FBt4/detection_loss‚0.9977726311760616 _runtime‚1660.237863055 .loss/FRt2/detection_loss‚2.950954821801955 .loss/FRt3/detection_loss‚2.444185842573643¦NÏ•Ô’Æ‚(:‡ ð‹ãÔûô·® gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.907' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7814 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º«A»uÇ‚(:ê ð‹ãÔæ·û° disk.sda3.in‚0.046875! disk.nvme0c0n1.in‚ 334.1484375 cpu‚25.260790961226466 disk.sda.out‚ 4185.7734375 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0& disk.nvme0n1.out‚18426.24267578125$ disk./.usageGB‚260.7504234313965( disk.nvme0c0n1.out‚18426.24267578125 disk.sda.in‚0.046875 disk.drbd1716.in‚0 disk.dm-1.in‚0.046875 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.rbd1.out‚ 7.5146484375 disk.nvme0n1.in‚ 334.1484375 disk.drbd2266.in‚0 disk.drbd2080.out‚0) proc.memory.availableMB‚ 1665.32421875 network.sent‚ 3.841675e+06* disk./.usagePercent‚17.499179700658786 disk.dm-0.in‚ 334.1484375 disk.sda2.in‚0 disk.rbd0.in‚0$ memory_percent‚89.83566761016846# disk.dm-0.out‚18426.24267578125 disk.rbd1.in‚0 disk.drbd2080.in‚0# proc.memory.rssMB‚ 2676.64453125 proc.cpu.threads‚17 network.recv‚639390 disk.sda3.out‚ 4185.7734375 disk.drbd1716.out‚0 disk.dm-1.out‚ 4185.7734375* proc.memory.percent‚16.336941719055176B’ÞÈ‚(:’ ÿ‹ãÔ´ž·® gpu.0.gpu‚29.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚102.732' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.546400000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JÏsÉ‚(:è ÿ‹ãÔù®þ± disk.sda.out‚ 4232.8515625 disk.sda2.in‚0 disk.sda2.out‚0 disk.sda3.in‚0.046875 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda.in‚0.046875 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.dm-0.in‚ 334.1484375! disk.nvme0c0n1.in‚ 334.1484375# memory_percent‚89.8357629776001( disk.nvme0c0n1.out‚18426.51611328125& disk.nvme0n1.out‚18426.51611328125$ disk./.usageGB‚260.7504234313965 disk.dm-1.in‚0.046875 disk.rbd0.out‚0# proc.memory.rssMB‚ 2676.64453125 network.sent‚ 3.950179e+06 disk.drbd1716.out‚0 disk.sda3.out‚ 4232.8515625 cpu‚25.25919807345779* disk./.usagePercent‚17.499179700658786 disk.dm-1.out‚ 4232.8515625 disk.rbd0.in‚0# disk.dm-0.out‚18426.51611328125 disk.rbd1.out‚ 7.5146484375* proc.memory.percent‚16.336941719055176 network.recv‚646000 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 334.1484375) proc.memory.availableMB‚ 1665.30859375@ô1’Ê‚(:‡ ŽŒãÔÚÒÊ® gpu.0.gpu‚20.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2658 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï•,pË‚(:å ŽŒãÔâ¦Ò± disk.dm-1.out‚ 4277.7421875 disk.drbd2265.out‚0 disk.drbd2080.out‚0 network.recv‚649530 disk.nvme0n1.in‚ 334.1484375 disk.dm-1.in‚0.046875 disk.sda.out‚ 4277.7421875 disk.drbd2266.in‚0 disk.drbd2266.out‚0$ memory_percent‚89.83423709869385) proc.memory.availableMB‚ 1665.55859375 proc.cpu.threads‚17* disk./.usagePercent‚17.499183796779334 disk.rbd1.out‚ 7.6630859375 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.in‚0.046875 disk.drbd2080.in‚0! disk.dm-0.out‚18427.541015625# proc.memory.rssMB‚ 2676.64453125* proc.memory.percent‚16.336941719055176 disk.sda.in‚0.046875 disk.sda3.out‚ 4277.7421875& disk.nvme0c0n1.out‚18427.541015625$ disk.nvme0n1.out‚18427.541015625 disk.drbd1716.out‚0% disk./.usageGB‚260.75048446655273 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 334.1484375 disk.drbd1716.in‚0 disk.sda1.out‚0 network.sent‚ 3.955597e+06 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.dm-0.in‚ 334.1484375 disk.sda1.in‚0 cpu‚25.257239301312783õêK…2Ì- _wandbruntime‚1699  _runtime‚1699Pâu ÈÍ _runtime‚1699.384668689 _step‚167 ' lr-AdamW/pg1‚0.00021145720016563696 ' lr-AdamW/pg2‚0.00021145720016563696 trainer/global_step‚2099 $ _timestamp‚1.788397074747786e+09ÌÚÍÎÇ ' lr-AdamW/pg2‚0.00021145720016563696 trainer/global_step‚2099 $ _timestamp‚1.788397074747786e+09 _runtime‚1699.384668689 _step‚167 ' lr-AdamW/pg1‚0.00021145720016563696§ÒÏ2Ï- _wandbruntime‚1699  _runtime‚1699¯PhÇ'ÐÁO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 2939163301267 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221942784 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚27740420 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚55618340711424 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚515 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 % _timestamp‚1.7883970747485554e+09 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚55618357750784 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚58557338891295 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 2938998179871 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚42154645 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚69442204 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚64962354108416 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚61963428940800 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27267521 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚58466817854464 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚2099 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27267523 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42174681 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198002688 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2998949107712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚58466835857408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15909404 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15909406 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69422166 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2947473337344 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61414291191808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58466835857408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2947642336256 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚58557521052051 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2947473337344 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚61414478193664 _runtime‚1699.385699044 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42174681 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11831014 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚61414478193664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42154645 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27267521 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58466817854464 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2947642336256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11830499 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _step‚168 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64962576051200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚61414291191808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚517 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚58466817854464 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚2099 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27267523 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚42174681 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198002688 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 2998949107712 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚58466835857408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚15909404 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚15909406 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚69422166 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 2947473337344 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚61414291191808 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚58466835857408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 2947642336256 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚58557521052051 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 2947473337344 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚61414478193664 _runtime‚1699.385699044 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚42174681 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚11831014 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚61414478193664 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚42154645 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27267521 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚58466817854464 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 2947642336256 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚11830499 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _step‚168 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚64962576051200 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚61414291191808 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚517 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2948866475520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15916781 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚64992687073792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2948866475520 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚61992341889024 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42174432 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚55644300933120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚501 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚61443140709888 K>DeviceStatsMonitor.n«¥non_train_batch_end/allocation.all.allocated‚69474676 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2940387344415 _runtime‚1700.140399548 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27280180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27280182 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27280180 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230318080 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58584853399139 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚61992317948928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚61443140709888 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2940552466019 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61442953694720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3000369124864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚58494105222144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27753379 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚55644283893760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27752878 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚499 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206377984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚2099 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 2948866475520 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚15916781 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚64992687073792 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 2948866475520 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚61992341889024 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚42174432 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚55644300933120 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚501 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚61443140709888 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚69474676 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 2940387344415 _runtime‚1700.140399548 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27280180 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚27280182 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27280180 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230318080 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚58584853399139 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚61992317948928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚61443140709888 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 2940552466019 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚61442953694720 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3000369124864 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚58494105222144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚27753379 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚55644283893760 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚27752878 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚499 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206377984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 trainer/global_step‚2099 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59924151987259 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚62847617753088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43147347 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3008469998279 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3008635119675 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _step‚172 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12103742 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28381998 % _timestamp‚1.7883971148111775e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27900473 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28381458 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚59830286091264 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚59830286091264 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71047818 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚56915499828224 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43167383 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚59830304094208 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63409076352000 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62847430751232 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59923969826503 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚63409052411904 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214779904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3017144659968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3017313658880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27900471 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56915516867584 G?De…ÚKRviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚66478963682304 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27900471 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71067856 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚59830304094208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16278256 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16278254 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚66479202402304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚59924151987259 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚62847617753088 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚43147347 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3008469998279 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3008635119675 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _step‚172 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12103742 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚28381998 % _timestamp‚1.7883971148111775e+09 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚27900473 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚28381458 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚59830286091264 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚59830286091264 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚71047818 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚56915499828224 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚43167383 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚59830304094208 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚63409076352000 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚62847430751232 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚59923969826503 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚63409052411904 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214779904 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3017144659968 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3017313658880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚27900471 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚56915516867584 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚66478963682304 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚27900471 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚71067856 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚59830304094208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16278256 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16278254 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚66479202402304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3018706810368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3018706810368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚549 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16285633 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27913132 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16285631 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚63437989300224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43187196 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59857573458944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208475136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3009859162823 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71100328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71100328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56941460049920 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43167134 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232415232 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚63437965360128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚173 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43187196 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28394284 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚62876280269312 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚66509296835072 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43167134 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56941443010560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚2149 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961]}’§#æÏM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚27913130 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚28394835 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚59857555456000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3018537798144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚551 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12108653 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3071331474944 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚27913130 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3071539950080 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3018706810368 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3018706810368 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚549 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16285633 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚27913132 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16285631 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚63437989300224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚43187196 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚59857573458944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208475136 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3009859162823 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚71100328 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚71100328 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚56941460049920 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚43167134 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232415232 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚63437965360128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _step‚173 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚43187196 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚28394284 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚62876280269312 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚66509296835072 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚43167134 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚56941443010560 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚2149 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961­åÎ2ç- _wandbruntime‚1740  _runtime‚1740à½ÉSèÍ /loss/FBt2/detection_loss‚2.6276805121451616 /loss/FBt3/detection_loss‚3.5777341723442078 $ _timestamp‚1.788397115557023e+09 .loss/FRt2/detection_loss‚2.887917666724234 1loss/SRqqt1/assignment_loss‚1.676560303704305 1loss/SRqqt4/assignment_loss‚1.453310499397608 1loss/FBt4/assignment_loss‚0.19124057106305317 /loss/FRt1/detection_loss‚2.4059473957334245 /loss/FRt1/assignment_loss‚4.280319030795779 1loss/SRqqt3/assignment_loss‚1.855000619183887 /loss/FBt1/detection_loss‚2.3020288621385894 trainer/global_step‚2149 1loss/SRqqt4/detection_loss‚1.3070559348051365 2loss/SRqqt2/assignment_loss‚1.8603954728354106 0loss/FBt1/assignment_loss‚0.6057984779278437 0loss/SRqqt2/detection_loss‚2.354614119638096 _step‚174 .loss/FRt4/detection_loss‚1.957685628147037 _runtime‚ 1740.19390197 /loss/FBt4/detection_loss‚0.6915053083868599 /loss/FRt3/assignment_loss‚2.688533849788435 0loss/FRt4/assignment_loss‚1.9086114521379824 /loss/FRt2/assignment_loss‚4.924361619082364 0loss/FBt2/assignment_loss‚0.6198155232059435 0loss/FBt3/assignment_loss‚0.7516485452651978 /loss/FRt3/detection_loss‚2.5570051155307074 %loss/total_loss‚49.63191732693064 0loss/SRqqt1/detection_loss‚2.066733615642244 epoch‚1 0loss/SRqqt3/detection_loss‚2.080413031307134ª“E·XéÒ 2loss/SRqqt2/assignment_loss‚1.8603954728354106 0loss/FBt1/assignment_loss‚0.6057984779278437 0loss/SRqqt2/detection_loss‚2.354614119638096 _step‚174 .loss/FRt4/detection_loss‚1.957685628147037 _runtime‚ 1740.19390197 /loss/FBt4/detection_loss‚0.6915053083868599 /loss/FRt3/assignment_loss‚2.688533849788435 0loss/FRt4/assignment_loss‚1.9086114521379824 /loss/FRt2/assignment_loss‚4.924361619082364 0loss/FBt2/assignment_loss‚0.6198155232059435 0loss/FBt3/assignment_loss‚0.7516485452651978 /loss/FRt3/detection_loss‚2.5570051155307074 %loss/total_loss‚49.63191732693064 0loss/SRqqt1/detection_loss‚2.066733615642244 epoch‚1 0loss/SRqqt3/detection_loss‚2.080413031307134 /loss/FBt2/detection_loss‚2.6276805121451616 /loss/FBt3/detection_loss‚3.5777341723442078 $ _timestamp‚1.788397115557023e+09 .loss/FRt2/detection_loss‚2.887917666724234 1loss/SRqqt1/assignment_loss‚1.676560303704305 1loss/SRqqt4/assignment_loss‚1.453310499397608 1loss/FBt4/assignment_loss‚0.19124057106305317 /loss/FRt1/detection_loss‚2.4059473957334245 /loss/FRt1/assignment_loss‚4.280319030795779 1loss/SRqqt3/assignment_loss‚1.855000619183887 /loss/FBt1/detection_loss‚2.3020288621385894 trainer/global_step‚2149 1loss/SRqqt4/detection_loss‚1.3070559348051365®c¼òê‚(:’ ÊŒãÔˆ™¨® gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.841' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.568199999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\Âvë‚(:ë ÊŒãÔ£›à° disk.sda1.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.dm-0.out‚18428.7314453125 disk.sda3.in‚ 0.05078125 disk.drbd1716.out‚0) proc.memory.availableMB‚ 1664.80859375# proc.memory.rssMB‚ 2676.64453125 network.sent‚ 4.17224e+06 network.recv‚670304$ disk./.usageGB‚260.7504348754883 disk.rbd1.in‚0 disk.sda.in‚ 0.05078125 disk.drbd2266.in‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 4454.9765625* proc.memory.percent‚16.336941719055176% disk.nvme0n1.out‚18428.7314453125 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda3.out‚ 4454.9765625' disk.nvme0c0n1.out‚18428.7314453125 disk.nvme0n1.in‚ 334.1484375 disk.dm-1.in‚ 0.05078125 disk.drbd2265.out‚0# memory_percent‚89.8388147354126 disk.sda1.in‚0 disk.sda.out‚ 4454.9765625 disk.drbd2265.in‚0 proc.cpu.threads‚17) disk./.usagePercent‚17.49918046868139 disk.dm-0.in‚ 334.1484375 disk.rbd1.out‚ 7.75830078125! disk.nvme0c0n1.in‚ 334.1484375 cpu‚25.250560322056774$ž‘ì‚(:† ÙŒãÔÇî­® gpu.0.gpu‚20.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.865' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.373 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚039xxí‚(:í ÙŒãԥ߲ disk.drbd2265.out‚0) proc.memory.availableMB‚ 1664.55859375 disk.sda3.in‚ 0.05078125 disk.rbd1.in‚0 disk.sda.in‚ 0.05078125 proc.cpu.threads‚17 disk.dm-1.out‚ 4521.5390625 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 4521.5390625 disk.rbd1.out‚ 7.75830078125 disk.nvme0n1.in‚ 334.1484375# proc.memory.rssMB‚ 2676.64453125* proc.memory.percent‚16.336941719055176 network.recv‚676355 disk.sda2.out‚0 cpu‚25.248983624362825 network.sent‚ 4.270033e+06 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 334.1484375' disk.nvme0c0n1.out‚18428.9423828125% disk.nvme0n1.out‚18428.9423828125 disk.drbd1716.in‚0$ disk./.usageGB‚260.7504348754883 disk.drbd2266.in‚0" disk.dm-0.out‚18428.9423828125 disk.sda.out‚ 4521.5390625 disk.sda2.in‚0) disk./.usagePercent‚17.49918046868139 disk.dm-1.in‚ 0.05078125 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 334.1484375 disk.drbd1716.out‚0$ memory_percent‚89.84034061431885 disk.drbd2266.out‚0à#"ñ2î- _wandbruntime‚1779  _runtime‚1779@ývÉïà ' lr-AdamW/pg2‚0.00021062522253997542 trainer/global_step‚2199 % _timestamp‚1.7883971544673705e+09 _runtime‚1779.104254987 _step‚175 ' lr-AdamW/pg1‚0.00021062522253997542÷šÎðÈ trainer/global_step‚2199 % _timestamp‚1.7883971544673705e+09 _runtime‚1779.104254987 _step‚175 ' lr-AdamW/pg1‚0.00021062522253997542 ' lr-AdamW/pg2‚0.00021062522253997542¯xSÒ2ñ- _wandbruntime‚1779  _runtime‚1779 _ÆþÇ'òÁO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚1779.105622809 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3141018299904 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚61193754328064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214779904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3077941816687 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28533421 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238720000 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚61290782922467 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3086984981504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16647106 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16647104 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64854699823104 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28533423 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚58212675984384 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚58212658945024 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚44140049 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚64854723763200 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3140803520000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚72673470 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711040 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚67995503343104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚64280757312512 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚29021483 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚64280570310656 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚61193772331008 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7883971544682815e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚61193772331008 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚28533423 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚64280570310656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12373840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3078106938083 _step‚176 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28533421 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44160085 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12374377 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚2199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚61290600761711 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3086984981504 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚537 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67995742063104 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚64280570310656 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12373840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3078106938083 _step‚176 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚28533421 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚44160085 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12374377 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚2199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚61290600761711 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3086984981504 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚537 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚67995742063104 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚1779.105622809 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3141018299904 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚61193754328064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214779904 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3077941816687 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚28533421 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238720000 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚61290782922467 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3086984981504 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚16647106 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚16647104 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚64854699823104 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚28533423 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚58212675984384 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚58212658945024 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248°äÊ» 2ô- _wandbruntime‚1779  _runtime‚1779̸G©õËM REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44179898 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚68026069528064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212669440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3142220147200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3079496102835 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3088209120768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28546080 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚58238602127360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28546080 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3088378132992 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚68025832918528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3079330981231 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58238619166720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16654483 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚61317933108591 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚61221041695744 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236609536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚1779.847413011 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28546082 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚64309419828736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12379769 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44159836 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚61221041695744 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72725980 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚61318115269555 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29034252 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.f¥]€º(reed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3088378132992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44159836 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44179898 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚61221023692800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚61221023692800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3088209120768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29033728 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7883971552096825e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚64309419828736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64883612771328 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚64309232813568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16654481 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72725980 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12379247 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _step‚177 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72705916 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72705916 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64883636711424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 aRíÖ&öÐM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3088209120768 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29033728 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7883971552096825e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚64309419828736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚64883612771328 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚64309232813568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚16654481 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚72725980 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12379247 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _step‚177 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚72705916 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚72705916 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚64883636711424 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚44179898 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚68026069528064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212669440 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3142220147200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3079496102835 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3088209120768 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚28546080 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚58238602127360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚28546080 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3088378132992 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚68025832918528 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3079330981231 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚58238619166720 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚16654483 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚61317933108591 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚61221041695744 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236609536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚1779.847413011 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚28546082 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚64309419828736 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12379769 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚44159836 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚61221041695744 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚72725980 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚61318115269555 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29034252 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3088378132992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚44159836 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚44179898 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚61221023692800 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚61221023692800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0±½"G2÷- _wandbruntime‚1779  _runtime‚1779w•6ZøÔ 0loss/FRt4/assignment_loss‚2.0441121068494073 /loss/FRt3/detection_loss‚2.6312907824390814 1loss/SRqqt3/detection_loss‚2.1117123196070846 &loss/total_loss‚52.148829765693776 epoch‚1 _step‚178 2loss/SRqqt4/assignment_loss‚1.2346704048701587 /loss/FRt2/detection_loss‚2.8342857770621777 0loss/SRqqt1/detection_loss‚2.097753635861657 /loss/FBt1/detection_loss‚2.4876166444558363 .loss/FBt2/detection_loss‚3.389219510555267 2loss/SRqqt1/assignment_loss‚1.8570332320576364 1loss/FBt4/assignment_loss‚0.36051806093259003 trainer/global_step‚2199 /loss/FRt4/detection_loss‚1.9884399227027236 % _timestamp‚1.7883971552116466e+09 0loss/FBt2/assignment_loss‚0.5997398060562773 1loss/SRqqt2/detection_loss‚2.6780933448762605 /loss/FBt4/detection_loss‚0.5798150345136711 .loss/FRt1/assignment_loss‚4.53841748102656 .loss/FRt1/detection_loss‚2.297937534890085 -loss/FBt3/detection_loss‚3.69433833360672 _runtime‚1779.848525381 0loss/FRt3/assignment_loss‚3.8652960720815157 2loss/SRqqt2/assignment_loss‚1.7092422952480388 1loss/FBt3/assignment_loss‚0.32422046065330506 1loss/SRqqt4/detection_loss‚1.1174501503507297 /loss/FRt2/assignment_loss‚4.562150988727808 2loss/SRqqt3/assignment_loss‚2.4261065538633955 0loss/FBt1/assignment_loss‚0.7193693124057833];Ì_ùÙ 1loss/FBt3/assignment_loss‚0.32422046065330506 1loss/SRqqt4/detection_loss‚1.1174501503507297 /loss/FRt2/assignment_loss‚4.562150988727808 2loss/SRqqt3/assignment_loss‚2.4261065538633955 0loss/FBt1/assignment_loss‚0.7193693124057833 0loss/FRt4/assignment_loss‚2.0441121068494073 /loss/FRt3/detection_loss‚2.6312907824390814 1loss/SRqqt3/detection_loss‚2.1117123196070846 &loss/total_loss‚52.148829765693776 epoch‚1 _step‚178 2loss/SRqqt4/assignment_loss‚1.2346704048701587 /loss/FRt2/detection_loss‚2.8342857770621777 0loss/SRqqt1/detection_loss‚2.097753635861657 /loss/FBt1/detection_loss‚2.4876166444558363 .loss/FBt2/detection_loss‚3.389219510555267 2loss/SRqqt1/assignment_loss‚1.8570332320576364 1loss/FBt4/assignment_loss‚0.36051806093259003 trainer/global_step‚2199 /loss/FRt4/detection_loss‚1.9884399227027236 % _timestamp‚1.7883971552116466e+09 0loss/FBt2/assignment_loss‚0.5997398060562773 1loss/SRqqt2/detection_loss‚2.6780933448762605 /loss/FBt4/detection_loss‚0.5798150345136711 .loss/FRt1/assignment_loss‚4.53841748102656 .loss/FRt1/detection_loss‚2.297937534890085 -loss/FBt3/detection_loss‚3.69433833360672 _runtime‚1779.848525381 0loss/FRt3/assignment_loss‚3.8652960720815157 2loss/SRqqt2/assignment_loss‚1.7092422952480388²q õ‘ú‚(:† èŒãԛ溮 gpu.0.gpu‚27.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.865' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.373 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§8Àdqû‚(:æ èŒãÔ™®Ÿ±# disk.nvme0n1.out‚18433.08984375 disk.rbd0.out‚0 network.recv‚680755 disk.sda.in‚ 0.05078125 disk.drbd1716.in‚0 disk.dm-1.out‚ 4556.6640625 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.dm-0.out‚18433.08984375( proc.memory.availableMB‚ 1665.0234375 proc.cpu.threads‚17 network.sent‚ 4.275931e+06 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.sda3.in‚ 0.05078125 disk.rbd1.in‚0$ memory_percent‚89.83750343322754 disk.dm-1.in‚ 0.05078125 disk.drbd2080.in‚0 disk.rbd1.out‚ 7.8212890625 cpu‚25.247360373448934* disk./.usagePercent‚17.499395771017685$ disk./.usageGB‚260.7536430358887 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 4556.6640625! disk.nvme0c0n1.in‚ 334.1484375 disk.sda2.in‚0 disk.dm-0.in‚ 334.1484375 disk.nvme0n1.in‚ 334.1484375# proc.memory.rssMB‚ 2676.64453125* proc.memory.percent‚16.336941719055176 disk.sda.out‚ 4556.6640625 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0% disk.nvme0c0n1.out‚18433.08984375 ·œÌü‚(:’ ÷ŒãÔÿÒ¤® gpu.0.gpu‚36.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.088' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.217599999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ûT$Éuý‚(:ê ÷ŒãÔÿ茲 disk.dm-0.in‚ 334.1484375 disk.dm-1.out‚ 4599.2265625 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda1.in‚0 cpu‚25.24571342432347 network.recv‚687182 disk.rbd1.out‚ 7.8212890625! disk.nvme0c0n1.in‚ 334.1484375' disk.nvme0c0n1.out‚18433.6455078125 disk.dm-1.in‚ 0.05078125# proc.memory.rssMB‚ 2676.64453125 proc.cpu.threads‚17$ disk./.usageGB‚260.7537384033203 disk.drbd2265.in‚0 disk.sda3.out‚ 4599.2265625 disk.sda.in‚ 0.05078125$ memory_percent‚89.83902931213379 disk.drbd2080.out‚0 disk.drbd1716.out‚0) disk./.usagePercent‚17.49940217120604 disk.rbd0.out‚0" disk.dm-0.out‚18433.6455078125% disk.nvme0n1.out‚18433.6455078125* proc.memory.percent‚16.336941719055176 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 network.sent‚ 4.384556e+06 disk.rbd0.in‚0 disk.sda3.in‚ 0.05078125 disk.nvme0n1.in‚ 334.1484375( proc.memory.availableMB‚ 1664.7734375 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 4599.2265625 disk.drbd2265.out‚0›;®E’þ‚(:‡ †ãÔëÔõ­ gpu.0.gpu‚20.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.973' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9946 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ͱDµxÿ‚(:í †ãÔ¹•¬±% disk./.usageGB‚260.75361251831055 disk.sda1.in‚0& disk.nvme0n1.out‚18434.66064453125 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.dm-0.in‚ 334.1484375 disk.rbd1.in‚0( proc.memory.availableMB‚ 1664.5234375* disk./.usagePercent‚17.499393722957414 disk.rbd1.out‚ 7.8212890625 disk.rbd0.out‚0* proc.memory.percent‚16.336941719055176 disk.sda.out‚ 4647.828125 disk.dm-1.out‚ 4647.828125 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0( disk.nvme0c0n1.out‚18434.66064453125 disk.nvme0n1.in‚ 334.1484375 network.recv‚690869 disk.dm-1.in‚ 0.05078125 disk.drbd2265.out‚0$ memory_percent‚89.84055519104004# proc.memory.rssMB‚ 2676.64453125 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda.in‚ 0.05078125 disk.drbd1716.out‚0# disk.dm-0.out‚18434.66064453125 disk.sda3.out‚ 4647.828125! disk.nvme0c0n1.in‚ 334.1484375 cpu‚25.244139362691577 disk.sda3.in‚ 0.05078125 network.sent‚ 4.389993e+06€´C®2€- _wandbruntime‚1818  _runtime‚18183¸÷µÇÁ % _timestamp‚1.7883971943074586e+09 _runtime‚1818.944366909 _step‚179 & lr-AdamW/pg1‚0.0002101769504520479 & lr-AdamW/pg2‚0.0002101769504520479 trainer/global_step‚2249@tÇŒÌ‚Æ & lr-AdamW/pg2‚0.0002101769504520479 trainer/global_step‚2249 % _timestamp‚1.7883971943074586e+09 _runtime‚1818.944366909 _step‚179 & lr-AdamW/pg1‚0.0002101769504520479³ôÃÚä2ƒ- _wandbruntime‚1818  _runtime‚1818ÓÎ’&„ÁO UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚65713896871936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚62557222564864 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29166371 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3211908487168 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3211687415808 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29166373 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45132751 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3147578756491 % _timestamp‚1.7883971943082438e+09 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚65713709870080 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚59509835101184 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚62657231696919 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65713709870080 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚66300347234304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17015956 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3156656304128 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3147413635095 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3211687415808 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29166373 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚45132751 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3147578756491 % _timestamp‚1.7883971943082438e+09 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚65713709870080 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚59509835101184 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚62657231696919 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚65713709870080 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚66300347234304 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17015956 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3156656304128 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3147413635095 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3211908487168 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396´¸Q)2†- _wandbruntime‚1819  _runtime‚18193·Ñ&‡ËM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45172600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚69542599353856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3148967921243 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚62584509932544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚65742559388160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3157880443392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚509 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚507 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204280832 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚62584491929600 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228220928 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29674159 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚62684564043799 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29179030 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29674668 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12651335 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74351632 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74351632 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚65742559388160 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45152538 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚69542371132928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29179032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12650828 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3148802799639 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29179030 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74331568 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3158049455616 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _step‚181 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3158049455616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199aŽç<Ö&ˆÐM TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚62584491929600 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228220928 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚29674159 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚62684564043799 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚29179030 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚29674668 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚12651335 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚74351632 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚74351632 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚65742559388160 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚45152538 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚69542371132928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚29179032 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12650828 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3148802799639 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚29179030 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚74331568 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3158049455616 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _step‚181 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3158049455616 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚45172600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚69542599353856 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3148967921243 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚62584509932544 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚65742559388160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3157880443392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚509 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚507 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204280832µ]ýœ’2‰- _wandbruntime‚1819  _runtime‚1819›&_WŠÑ 1loss/FBt1/assignment_loss‚0.38163391174524663 /loss/FRt2/detection_loss‚3.1541729774968377 0loss/SRqqt1/detection_loss‚2.188636113154261 -loss/FBt1/detection_loss‚2.74811420073876 /loss/FBt3/detection_loss‚3.5699299971262612 /loss/FBt4/detection_loss‚0.6093330849748381 _runtime‚1819.702303907 /loss/FBt2/detection_loss‚2.8328407804171243 /loss/FRt2/assignment_loss‚4.543248225902689 0loss/FRt4/assignment_loss‚2.1721150698485197 epoch‚1 2loss/SRqqt2/assignment_loss‚1.6802267776984794 1loss/SRqqt3/assignment_loss‚2.014610069684493 1loss/SRqqt4/assignment_loss‚1.507682828646567 0loss/FBt3/assignment_loss‚0.2400153527657191 0loss/SRqqt3/detection_loss‚2.209549455306469 1loss/SRqqt4/detection_loss‚1.0740042960064278 % _timestamp‚1.7883971950655189e+09 /loss/FRt3/assignment_loss‚4.130092741445053 /loss/FRt1/detection_loss‚2.2375044604627097 0loss/SRqqt2/detection_loss‚1.916349243883993 &loss/total_loss‚50.244813592012875 trainer/global_step‚2249 /loss/FRt1/assignment_loss‚4.201105731289561 2loss/SRqqt1/assignment_loss‚1.7297039161387242 1loss/FBt2/assignment_loss‚0.20372207894680766 .loss/FRt3/detection_loss‚2.539419197758963 0loss/FBt4/assignment_loss‚0.3062887807721651 /loss/FRt4/detection_loss‚2.0545142998022063 _step‚182.êe‹Ö /loss/FRt3/assignment_loss‚4.130092741445053 /loss/FRt1/detection_loss‚2.2375044604627097 0loss/SRqqt2/detection_loss‚1.916349243883993 &loss/total_loss‚50.244813592012875 trainer/global_step‚2249 /loss/FRt1/assignment_loss‚4.201105731289561 2loss/SRqqt1/assignment_loss‚1.7297039161387242 1loss/FBt2/assignment_loss‚0.20372207894680766 .loss/FRt3/detection_loss‚2.539419197758963 0loss/FBt4/assignment_loss‚0.3062887807721651 /loss/FRt4/detection_loss‚2.0545142998022063 _step‚182 1loss/FBt1/assignment_loss‚0.38163391174524663 /loss/FRt2/detection_loss‚3.1541729774968377 0loss/SRqqt1/detection_loss‚2.188636113154261 -loss/FBt1/detection_loss‚2.74811420073876 /loss/FBt3/detection_loss‚3.5699299971262612 /loss/FBt4/detection_loss‚0.6093330849748381 _runtime‚1819.702303907 /losΤÜ÷s/FBt2/detection_loss‚2.8328407804171243 /loss/FRt2/assignment_loss‚4.543248225902689 0loss/FRt4/assignment_loss‚2.1721150698485197 epoch‚1 2loss/SRqqt2/assignment_loss‚1.6802267776984794 1loss/SRqqt3/assignment_loss‚2.014610069684493 1loss/SRqqt4/assignment_loss‚1.507682828646567 0loss/FBt3/assignment_loss‚0.2400153527657191 0loss/SRqqt3/detection_loss‚2.209549455306469 1loss/SRqqt4/detection_loss‚1.0740042960064278 % _timestamp‚1.7883971950655189e+09¶ö‘Œ‚(:’ •ãÔÓé™® gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.311' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.462200000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆãz‚(:ï •ãÔ…Œ¦± disk.drbd2080.in‚0 disk.dm-0.in‚ 334.1484375 disk.sda3.in‚ 0.05078125 network.sent‚ 4.395918e+06 disk.drbd2265.in‚0& disk.nvme0n1.out‚18434.93408203125 disk.sda2.out‚0 disk.drbd1716.in‚0# memory_percent‚89.8395299911499) proc.memory.availableMB‚ 1664.69140625* proc.memory.percent‚16.336941719055176% disk./.usageGB‚260.75361251831055 disk.rbd0.in‚0 disk.nvme0n1.in‚ 334.1484375 disk.dm-1.out‚ 4691.6328125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 proc.cpu.threads‚17* disk./.usagePercent‚17.499393722957414 disk.rbd1.in‚0 disk.rbd1.out‚ 7.8212890625 disk.sda2.in‚0 disk.drbd2080.out‚0 cpu‚25.24270520192378 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 4691.6328125( disk.nvme0c0n1.out‚18434.93408203125 disk.dm-1.in‚ 0.05078125# proc.memory.rssMB‚ 2676.64453125 disk.sda.in‚ 0.05078125 disk.sda.out‚ 4691.6328125! disk.nvme0c0n1.in‚ 334.1484375# disk.dm-0.out‚18434.93408203125 disk.sda1.out‚0 network.recv‚695309 disk.drbd2265.out‚0a‹u˜’Ž‚(:‡ ¤ãÔ‰À ® gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚103.382' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.6764 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0qê'Ïu‚(:ê ¤ãÔñÉÖ±' disk.nvme0c0n1.out‚18435.2060546875 disk.nvme0n1.in‚ 334.1484375 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 4734.796875 disk.drbd2266.in‚0" disk.dm-0.out‚18435.2060546875 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1664.69921875* disk./.usagePercent‚17.499393722957414% disk./.usageGB‚260.75361251831055 disk.sda.out‚ 4734.796875 disk.drbd2265.out‚0 network.recv‚701280 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 334.1484375! disk.nvme0c0n1.in‚ 334.1484375 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 4734.796875$ memory_percent‚89.83948230743408# proc.memory.rssMB‚ 2676.64453125* proc.memory.percent‚16.336941719055176 cpu‚25.24101548224299 disk.dm-1.in‚ 0.05078125 disk.sda3.in‚ 0.05078125 disk.rbd1.in‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 7.8212890625 network.sent‚ 4.504196e+06 disk.drbd1716.out‚0 disk.sda.in‚ 0.05078125 disk.drbd2080.out‚0% disk.nvme0n1.out‚18435.2060546875 disk.drbd2265.in‚0B©i@2- _wandbruntime‚1858  _runtime‚1858²P3¾É‘à _runtime‚1858.019909977 _step‚183 ' lr-AdamW/pg1‚0.00020970727924669334 ' lr-AdamW/pg2‚0.00020970727924669334 trainer/global_step‚2299 % _timestamp‚1.7883972333830388e+09Ø„§Î’È ' lr-AdamW/pg1‚0.00020970727924669334 ' lr-AdamW/pg2‚0.00020970727924669334 trainer/global_step‚2299 % _timestamp‚1.7883972333830388e+09 _runtime‚1858.019909977 _step‚183·6Þð 2“- _wandbruntime‚1858  _runtime‚1858ÛD{£Ç'”ÁO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚64023862632127 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3226158627840 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚67147036431360 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚30302197 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622848 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3216885453503 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67746018585600 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚67147036431360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚63920690801664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29799323 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75944812 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3282821386752 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚184 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29799323 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30302708 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17384806 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17384804 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12917902 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3282608704000 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚67146849429504 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29799321 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚60806977178624 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75924774 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚63920708804608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚511 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682752 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3226327626752 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3226158627840 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71028839972352 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚509 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12917393 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚63920708804608 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚60806994217984 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚63920690801664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64024044792883 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3217050574899 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚1858.020901539 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67745994645504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚71028603349504 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚67146849429504 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3226327626752 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚2299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46125453 % _timestamp‚1.7883972333837996e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29799321 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46125453 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚67146849429504 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3226327626752 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚2299 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚46125453 % _timestamp‚1.7883972333837996e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245249 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚29799321 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚46125453 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3216885453503 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚67746018585600 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚67147036431360 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚63920690801664 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚29799323 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚75944812 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3282821386752 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _step‚184 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚29799323 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚30302708 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17384806 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17384804 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚12917902 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3282608704000 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚67146849429504 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚29799321 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚60806977178624 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚75924774 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚63920708804608 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚511 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682752 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3226327626752 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3226158627840 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚71028839972352 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚509 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚12917393 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚63920708804608 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚60806994217984 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚63920690801664 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚64024044792883 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3217050574899 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚1858.020901539 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚67745994645504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0¸ªG[Õ2–- _wandbruntime‚1858  _runtime‚1858W´šÒЗËM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3227551766016 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚67175698947584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚63947978169344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202183680 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚60832937400320 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3218274618047 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3227720778240 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚494 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46145240 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚64051194979007 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _runtime‚1858.760944004 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71059164949504 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226123776 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75957220 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3227720778240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17392183 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3284031232000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30315335 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3284233415680 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 % _timestamp‚1.7883972341233678e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67774907593728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30314839 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17392181 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12922658 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71058938825728 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75977284 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46165302 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75957220 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29811982 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚46165302 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚75957220 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚29811982 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚185 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3227551766016 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚67175698947584 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚63947978169344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202183680 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚60832937400320 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3218274618047 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3227720778240 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚494 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚46145240 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚64051194979007 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _runtime‚1858.760944004 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚71059164949504 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226123776 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚75957220 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3227720778240 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚17392183 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3284031232000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30315335 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3284233415680 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 % _timestamp‚1.7883972341233678e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚67774907593728 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30314839 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17392181 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚12922658 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚71058938825728 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚75977284 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2¹OëÖn2™- _wandbruntime‚1858  _runtime‚1858±µÔTšÎ 0loss/FBt2/assignment_loss‚0.3765502000064421 /loss/FRt2/detection_loss‚2.6100500317720265 /loss/FBt4/detection_loss‚0.8655964340759625 %loss/total_loss‚53.86596342157573 % _timestamp‚1.7883972341252322e+09 /loss/FRt4/detection_loss‚1.8797219205720752 1loss/SRqqt4/detection_loss‚1.2531777491440643 _runtime‚1858.762030976 1loss/SRqqt3/assignment_loss‚1.787948467229542 0loss/FBt4/assignment_loss‚0.3058670953300739 0loss/SRqqt2/detection_loss‚2.665073875337839 2loss/SRqqt4/assignment_loss‚1.5699482494713486 0loss/FBt3/assignment_loss‚0.7143151164054871 1loss/SRqqt3/detection_loss‚1.9474758730645765 .loss/FBt1/detection_loss‚2.247644084195296 epoch‚1 trainer/global_step‚2299 0loss/FRt2/assignment_loss‚4.4210806626539965 0loss/FRt4/assignment_loss‚1.8554610528197943 0loss/FBt1/assignment_loss‚0.8286160038454303 /loss/FRt1/detection_loss‚2.3066783968521203 0loss/SRqqt1/detection_loss‚2.067221132516861 ,loss/FBt3/detection_loss‚7.0965735912323 -loss/FRt1/assignment_loss‚4.1346104948417 /loss/FRt3/detection_loss‚2.4811207891964333 /loss/FBt2/detection_loss‚3.1834525236716638 /loss/FRt3/assignment_loss‚3.731835190842791 _step‚186 2loss/SRqqt1/assignment_loss‚1.6999433796107768 2loss/SRqqt2/assignment_loss‚1.8360011068871245}“?…Y›Ó 2loss/SRqqt1/assignment_loss‚1.6999433796107768 2loss/SRqqt2/assignment_loss‚1.8360011068871245 0loss/FBt2/assignment_loss‚0.3765502000064421 /loss/FRt2/detection_loss‚2.6100500317720265 /loss/FBt4/detection_loss‚0.8655964340759625 %loss/total_loss‚53.86596342157573 % _timestamp‚1.7883972341252322e+09 /loss/FRt4/detection_loss‚1.8797219205720752 1loss/SRqqt4/detection_loss‚1.2531777491440643 _runtime‚1858.762030976 1loss/SRqqt3/assignment_loss‚1.787948467229542 0loss/FBt4/assignment_loss‚0.3058670953300739 0loss/SRqqt2/detection_loss‚2.665073875337839 2loss/SRqqt4/assignment_loss‚1.5699482494713486 0loss/FBt3/assignment_loss‚0.7143151164054871 1loss/SRqqt3/detection_loss‚1.9474758730645765 .loss/FBt1/detection_loss‚2.247644084195296 epoch‚1 trainer/global_step‚2299 0loss/FRt2/assignment_loss‚4.4210806626539965 0loss/FRt4/assignment_loss‚1.8554610528197943 0loss/FBt1/assignment_loss‚0.8286160038454303 /loss/FRt1/detection_loss‚2.3066783968521203 0loss/SRqqt1/detection_loss‚2.067221132516861 ,loss/FBt3/detection_loss‚7.0965735912323 -loss/FRt1/assignment_loss‚4.1346104948417 /loss/FRt3/detection_loss‚2.4811207891964333 /loss/FBt2/detection_loss‚3.1834525236716638 /loss/FRt3/assignment_loss‚3.731835190842791 _step‚186ºZ, œœ‚(:‘ ³ãÔ¾ùŽ® gpu.0.gpu‚9.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚123.221' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.644199999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04 Ðx‚(:í ³ãÔ§ü¨± disk.drbd2266.out‚0 disk.sda.in‚ 0.05078125 disk.rbd0.in‚0 disk.rbd0.out‚0* disk./.usagePercent‚17.499397563070428 disk.dm-1.in‚ 0.05078125 disk.drbd2265.in‚0 disk.sda3.in‚ 0.05078125 disk.rbd1.out‚ 7.9072265625 disk.drbd2080.out‚0 disk.dm-0.in‚ 334.1484375% disk.nvme0n1.out‚18435.4287109375* proc.memory.percent‚16.336941719055176 disk.drbd2080.in‚0 network.sent‚ 4.510444e+06 network.recv‚705865" disk.dm-0.out‚18435.4287109375 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 334.1484375 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 334.1484375$ memory_percent‚89.84260559082031 cpu‚25.239372322452017' disk.nvme0c0n1.out‚18435.4287109375 disk.drbd1716.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 4785.44921875 disk.sda.out‚ 4785.44921875% proc.memory.availableMB‚ 1664.1875# proc.memory.rssMB‚ 2676.64453125 proc.cpu.threads‚17% disk./.usageGB‚260.75366973876953 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 4785.44921875ú°’ž‚(:‡ Âãԡ᪮ gpu.0.gpu‚20.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7332 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Df­){Ÿ‚(:ð ÂãԢ㱠disk.rbd0.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 7.9697265625! disk.nvme0c0n1.in‚ 334.1484375& disk.nvme0n1.out‚18435.81103515625 network.recv‚711190 disk.sda1.in‚0* proc.memory.percent‚16.336941719055176$ disk./.usageGB‚260.7535705566406 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 4842.50390625( disk.nvme0c0n1.out‚18435.81103515625 disk.sda.in‚ 0.05078125 disk.drbd2080.in‚0 disk.sda3.in‚ 0.05078125 disk.rbd1.in‚0$ memory_percent‚89.83983993530273 cpu‚25.23813503584349 network.sent‚ 4.618189e+06 disk.drbd1716.in‚0 disk.drbd2080.out‚0 proc.cpu.threads‚17 disk.drbd1716.out‚0* disk./.usagePercent‚17.499390906874535 disk.sda.out‚ 4842.50390625 disk.dm-1.in‚ 0.05078125 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 334.1484375' proc.memory.availableMB‚ 1664.640625 disk.drbd2266.in‚0 disk.dm-0.in‚ 334.1484375# disk.dm-0.out‚18435.81103515625# proc.memory.rssMB‚ 2676.64453125 disk.sda2.in‚0 disk.dm-1.out‚ 4842.50390625àôT’ ‚(:‡ ÑãÔ–í“® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2658 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0m N~¡‚(:ó ÑãÔŸÖî± disk.rbd0.out‚0 disk.sda3.out‚ 4902.69140625 cpu‚25.236651835153776 disk.sda.in‚ 0.05078125 disk.drbd1716.out‚0 disk.drbd2265.in‚0! disk.nvme0c0n1.in‚ 334.1484375( disk.nvme0c0n1.out‚18436.02587890625$ disk./.usageGB‚260.7535705566406 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0 network.sent‚ 4.624226e+06 disk.sda.out‚ 4902.69140625 disk.drbd2266.in‚0 disk.nvme0n1.in‚ 334.1484375 network.recv‚715812 disk.sda2.in‚0 disk.drbd2265.out‚0# disk.dm-0.out‚18436.02587890625$ memory_percent‚89.83981609344482# proc.memory.rssMB‚ 2676.64453125 proc.cpu.threads‚17 disk.rbd1.in‚0& disk.nvme0n1.out‚18436.02587890625) proc.memory.availableMB‚ 1664.64453125 disk.rbd1.out‚ 7.9697265625 disk.dm-1.out‚ 4902.69140625 disk.drbd2080.in‚0 disk.dm-0.in‚ 334.1484375 disk.sda1.out‚0 disk.sda3.in‚ 0.05078125* disk./.usagePercent‚17.499390906874535 disk.dm-1.in‚ 0.05078125 disk.drbd2080.out‚0 disk.sda1.in‚0* proc.memory.percent‚16.336941719055176 disk.drbd1716.in‚0’¼‡€2¢- _wandbruntime‚1897  _runtime‚1897ý>¥Å£¿ % lr-AdamW/pg1‚0.000209216306802687 % lr-AdamW/pg2‚0.000209216306802687 trainer/global_step‚2349 % _timestamp‚1.7883972733323684e+09 _runtime‚1897.969332285 _step‚187aÅÊ¤Ä % lr-AdamW/pg1‚0.000209216306802687 % lr-AdamW/pg2‚0.000209216306802687 trainer/global_step‚2349 % _timestamp‚1.7883972733323684e+09 _runtime‚1897.969332285 _step‚187»i}íš2¥- _wandbruntime‚1897  _runtime‚1897ž[6Å'¦¿O TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17753656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚65284177041408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47138191 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚72545460303360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68579988988928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚566 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3286357271911 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77570464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30432271 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13190327 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚65390493567335 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2349 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚564 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13189763 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚1897.970536879 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47118155 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚65284177041408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚69191642056704 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚65390675728091 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚68580175990784 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚65284159038464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30432271 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 _step‚188 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17753654 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚62104136295424 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚68580175990784 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3295829950464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚68579988988928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3286522393307 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3295998949376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚65284159038464 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622848 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚69191665996800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0?ƒ£ê §ÄO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3295829950464 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 230548992 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚68579988988928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3286522393307 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3295998949376 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚65284159038464 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622848 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚69191665996800 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚17753656 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚65284177041408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚47138191 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚72545460303360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚68579988988928 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚566 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3286357271911 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚77570464 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚30432271 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚13190327 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚65390493567335 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚2349 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚564 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13189763 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚1897.970536879 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚47118155 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚65284177041408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚69191642056704 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚65390675728091 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚68580175990784 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚65284159038464 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚30432271 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 _step‚188 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚17753654 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚62104136295424 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚68580175990784 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199¼[ä2¨- _wandbruntime‚1898  _runtime‚1898Y9ä…Ñ&©ËM R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚68608651491840 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30956164 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77602936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3297223088640 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3287911558059 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚65311428403200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30444932 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13195133 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚65418008075179 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚189 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47137942 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3355211066368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7883972740845428e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77582872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47158004 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚65311428403200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3297392100864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚546 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚17761031 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _runtime‚1898.722081943 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2349 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚68608838507008 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚72575557596160 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚69220578945024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚62130096517120 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚65311446406144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30956710 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30444930 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚62130079477760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3297223088640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13195677 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77582872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚544 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3287746436455 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77602936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚69220555004928 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _runtime‚1898.722081943 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2349 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚68608838507008 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚72575557596160 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚69220578945024 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚62130096517120 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚65311446406144 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚30956710 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚30444930 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚62130079477760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3297223088640 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13195677 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚77582872 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚544 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3287746436455 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚77602936 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚69220555004928 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚30956164 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚77602936 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24399 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3297223088640 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3287911558059 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚65311428403200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚30444932 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚13195133 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚65418008075179 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚189 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚47137942 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3355211066368 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7883972740845428e+09 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚77582872 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚47158004 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20062 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚65311428403200 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3297392100864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24126 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚546½/“B2«- _wandbruntime‚1898  _runtime‚1898ú­ofX¬Ò 1loss/FBt2/assignment_loss‚0.48338380267068715 .loss/FRt3/detection_loss‚2.997277081012726 1loss/SRqqt3/detection_loss‚1.9321254353697708 .loss/FBt3/detection_loss‚5.117562850316365 trainer/global_step‚2349 _runtime‚1898.723663979 1loss/SRqqt3/assignment_loss‚2.061569559047135 _step‚190 /loss/FRt3/assignment_loss‚3.901593748367194 /loss/FRt2/detection_loss‚2.9407866736253103 0loss/FRt1/assignment_loss‚3.7333730616659486 /loss/FRt2/assignment_loss‚4.305354833602905 /loss/FBt2/detection_loss‚3.8396704494953156 % _timestamp‚1.7883972740868723e+09 2loss/SRqqt4/assignment_loss‚1.2279124808999209 0loss/FBt1/assignment_loss‚0.6742997678617636 /loss/FRt4/detection_loss‚1.7047183202772305 .loss/FBt4/detection_loss‚0.995589747767878 2loss/SRqqt1/assignment_loss‚1.7888151625792186 1loss/SRqqt1/detection_loss‚2.0022193676895563 &loss/total_loss‚54.556383576526414 2loss/SRqqt2/assignment_loss‚2.2035903988814933 0loss/FBt3/assignment_loss‚1.1788466941410054 1loss/FBt4/assignment_loss‚0.30017601750361994 0loss/SRqqt2/detection_loss‚2.362614928222284 .loss/FBt1/detection_loss‚2.868872140844663 epoch‚1 /loss/FRt4/assignment_loss‚2.440963860215812 .loss/FRt1/detection_loss‚2.249479480509488 1loss/SRqqt4/detection_loss‚1.2455877139591254`GÝ]­× /loss/FRt2/assignment_loss‚4.305354833602905 /loss/FBt2/detection_loss‚3.8396704494953156 % _timestamp‚1.7883972740868723e+09 2loss/SRqqt4/assignment_loss‚1.2279124808999209 0loss/FBt1/assignment_loss‚0.6742997678617636 /loss/FRt4/detection_loss‚1.7047183202772305 .loss/FBt4/detection_loss‚0.995589747767878 2loss/SRqqt1/assignment_loss‚1.7888151625792186 1loss/SRqqt1/detection_loss‚2.0022193676895563 &loss/total_loss‚54.556383576526414 2loss/SRqqt2/assignment_loss‚2.2035903988814933 0loss/FBt3/assignment_loss‚1.1788466941410054 1loss/FBt4/assignment_loss‚0.30017601750361994 0loss/SRqqt2/detection_loss‚2.362614928222284 .loss/FBt1/detection_loss‚2.868872140844663 epoch‚1 /loss/FRt4/assignment_loss‚2.440963860215812 .loss/FRt1/detection_loss‚2.249479480509488 1loss/SRqqt4/detection_loss‚1.2455877139591254 1loss/FBt2/assignment_loss‚0.48338380267068715 .loss/FRt3/detection_loss‚2.997277081012726 1loss/SRqqt3/detection_loss‚1.9321254353697708 .loss/FBt3/detection_loss‚5.117562850316365 trainer/global_step‚2349 _runtime‚1898.723663979 1loss/SRqqt3/assignment_loss‚2.061569559047135 _step‚190 /loss/FRt3/assignment_loss‚3.901593748367194 /loss/FRt2/detection_loss‚2.9407866736253103 0loss/FRt1/assignment_loss‚3.7333730616659486¾£>WÓ’®‚(:‡ àãÔ£Úÿ­ gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.419' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.0838 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ki¶¼r¯‚(:ç àãÔóÁ²° disk.drbd1716.in‚0 disk.drbd2266.out‚0# proc.memory.rssMB‚ 2676.64453125 network.sent‚ 4.630602e+06 network.recv‚720179) disk./.usagePercent‚17.49940729135673 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.sda3.out‚ 4957.1953125 disk.sda.in‚ 0.05078125 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 334.1484375 disk.sda.out‚ 4957.1953125$ disk./.usageGB‚260.7538146972656 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0( proc.memory.availableMB‚ 1664.3515625 disk.dm-1.out‚ 4957.1953125 disk.drbd2266.in‚0 disk.sda2.out‚0$ memory_percent‚89.84160423278809 disk.drbd2265.in‚0 disk.rbd0.out‚0! disk.dm-0.out‚18436.435546875$ disk.nvme0n1.out‚18436.435546875 disk.dm-0.in‚ 334.1484375 disk.sda3.in‚ 0.05078125 disk.rbd1.out‚ 8.001953125& disk.nvme0c0n1.out‚18436.435546875 disk.dm-1.in‚ 0.05078125 disk.drbd2080.out‚0 disk.nvme0n1.in‚ 334.1484375 disk.sda2.in‚0* proc.memory.percent‚16.336941719055176 cpu‚25.235241200603554 proc.cpu.threads‚17£ŸJ°‚(:’ ïãÔ´ÜŽ® gpu.0.gpu‚36.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚119.203' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.840600000000002 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’,áer±‚(:Ý ïãÔ±Šú° disk.nvme0n1.in‚ 334.1484375 disk.dm-1.out‚ 5016.5625$ memory_percent‚89.84003067016602' proc.memory.availableMB‚ 1664.609375# proc.memory.rssMB‚ 2676.64453125 disk.sda1.out‚0 disk.sda.in‚ 0.05078125 disk.drbd1716.out‚0* proc.memory.percent‚16.336941719055176 network.sent‚ 4.738191e+06 disk.rbd0.out‚0 disk.sda1.in‚0& disk.nvme0c0n1.out‚18436.685546875 disk.drbd1716.in‚0 disk.drbd2266.in‚0 network.recv‚726120* disk./.usagePercent‚17.499411131469742 disk.rbd1.in‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17 di•¶¿ösk.drbd2080.in‚0! disk.nvme0c0n1.in‚ 334.1484375$ disk.nvme0n1.out‚18436.685546875$ disk./.usageGB‚260.7538719177246 disk.dm-0.in‚ 334.1484375 disk.sda3.in‚ 0.05078125 disk.sda3.out‚ 5016.5625 disk.sda2.in‚0 cpu‚25.23379738935813 disk.rbd1.out‚ 8.001953125 disk.sda.out‚ 5016.5625 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.05078125 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0! disk.dm-0.out‚18436.685546875¡˜B’²‚(:‡ þãÔŽŸ¤® gpu.0.gpu‚30.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚118.377' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6754 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 W Ÿz³‚(:ï þãÔ£Æò° disk.rbd0.out‚0 disk.sda3.in‚ 0.05078125! disk.nvme0c0n1.in‚ 334.1484375 disk.sda2.in‚0 disk.drbd1716.in‚0$ memory_percent‚89.84034061431885 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 334.1484375 disk.sda.in‚ 0.05078125 disk.sda2.out‚0* proc.memory.percent‚16.336965560913086 cpu‚25.232392338268845 disk.nvme0n1.in‚ 334.1484375 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.dm-1.in‚ 0.05078125 disk.drbd2080.out‚0 disk.rbd1.out‚ 8.06494140625 disk.dm-1.out‚ 5063.12109375 network.sent‚ 4.743529e+06$ disk./.usageGB‚260.7538146972656 disk.sda1.in‚0' disk.nvme0c0n1.out‚18436.9990234375% disk.nvme0n1.out‚18436.9990234375 disk.sda.out‚ 5063.12109375) proc.memory.availableMB‚ 1664.55859375 proc.cpu.threads‚17 network.recv‚729612" disk.dm-0.out‚18436.9990234375 disk.sda3.out‚ 5063.12109375 disk.drbd1716.out‚0 disk.drbd2266.in‚0" proc.memory.rssMB‚ 2676.6484375) disk./.usagePercent‚17.49940729135673 disk.sda1.out‚0 disk.rbd1.in‚0 disk.drbd2265.in‚0q`9 2´- _wandbruntime‚1937  _runtime‚1937ÅXõzɵà _runtime‚1937.525912087 _step‚191 ' lr-AdamW/pg1‚0.00020870413543795036 ' lr-AdamW/pg2‚0.00020870413543795036 trainer/global_step‚2399 % _timestamp‚1.7883973128890083e+09ÔkÕÎ¶È _step‚191 ' lr-AdamW/pg1‚0.00020870413543795036 ' lr-AdamW/pg2‚0.00020870413543795036 trainer/global_step‚2399 % _timestamp‚1.7883973128890083e+09 _runtime‚1937.525912087¿ ê2·- _wandbruntime‚1937  _runtime‚1937® ögÆ'¸ÀO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚70637313408000 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31586126 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3424481301504 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3424696081408 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63401312451584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚66647645278208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3365670272000 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48130893 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3365670272000 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚70013128548352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚70637289467904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚66647645278208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚70013315550208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48110857 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13463077 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3365501273088 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79196116 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18122504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31065223 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48110857 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚66647627275264 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18122506 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214779904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238720000 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3355994211715 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3355829090319 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31065221 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚66757124502543 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 $ _timestamp‚1.788397312889943e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31585581 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79176078 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚66757306663299 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3365501273088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 trainer/global_step‚2399 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚1937.527262877 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31065223 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74062009489408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70013128548352 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756J¡>çË'¹ÅO Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚70013128548352 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160756 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚70637313408000 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912019968 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187001856 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚31586126 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3424481301504 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3424696081408 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚63401312451584 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚66647645278208 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3365670272000 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789961 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚48130893 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3365670272000 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚70013128548352 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚70637289467904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚66647645278208 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚70013315550208 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚48110857 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚13463077 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20036 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3365501273088 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚79196116 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18122504 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚31065223 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24126 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121396 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚48110857 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚66647627275264 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168998912 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18122506 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214779904 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238720000 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3355994211715 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3355829090319 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚31065221 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20036 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚66757124502543 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187001856 $ _timestamp‚1.788397312889943e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912019968 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚31585581 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚79176078 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168998912 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚66757306663299 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20038 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3365501273088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20038 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24126 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 trainer/global_step‚2399 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚1937.527262877 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24399 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚31065223 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚74062009489408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408À7ŽÌx2º- _wandbruntime‚1938  _runtime‚1938£cÑËó%»ËM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚66674896640000 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚2399 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214766592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79228588 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13469313 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48130644 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3367063423488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66784456849423 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238706688 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70041978066432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚70666226356224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63427238594560 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31598658 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74092100212224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70041978066432 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31077880 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3367063423488 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74092338918912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63427255633920 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79208524 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3366894411264 _step‚193 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3357218254863 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7883973136289005e+09 XJDeviceStatsMo~ƒ¼Þnitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199â»ì.Ö&¼ÐM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160964 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚63427238594560 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚31598658 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚74092100212224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015168 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚70041978066432 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚31077880 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3367063423488 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚74092338918912 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚63427255633920 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚79208524 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711040 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3366894411264 _step‚193 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245249 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121604 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3357218254863 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7883973136289005e+09 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚66674896640000 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚2399 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24399 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015168 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912019968 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214766592 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912019968 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711040 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚79228588 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 230548992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789961 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚13469313 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚48130644 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3367063423488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚66784456849423 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012224 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20062 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238706688 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚70041978066432 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚70666226356224ÁÌO¦b2½- _wandbruntime‚1938  _runtime‚1938 ‚tZ¾Ô /loss/FRt3/assignment_loss‚4.122945677489042 1loss/SRqqt2/detection_loss‚2.4332172724935743 0loss/SRqqt3/detection_loss‚2.254420119524002 %loss/total_loss‚51.66928065586248 epoch‚1 /loss/FRt4/assignment_loss‚1.949116973649888 /loss/FRt3/detection_loss‚2.7812960231676698 1loss/SRqqt4/detection_loss‚1.1448031397660574 /loss/FBt4/detection_loss‚0.8310364947737638 _step‚194 1loss/FBt4/assignment_loss‚0.28392823956245233 1loss/SRqqt2/assignment_loss‚2.463334016005198 1loss/FBt2/assignment_loss‚0.43586360447525296 .loss/FBt1/detection_loss‚2.389722916212949 .loss/FBt2/detection_loss‚3.356252211332321 trainer/global_step‚2399 /loss/FRt2/assignment_loss‚5.331000135790918 1loss/FBt3/assignment_loss‚0.22196779400110245 /loss/FRt1/detection_loss‚2.1985551334441977 1loss/SRqqt1/detection_loss‚2.0566542778375014 0loss/FBt1/assignment_loss‚0.8342177606920648 /loss/FRt2/detection_loss‚2.9523658983169065 /loss/FRt4/detection_loss‚1.7357939194355692 _runtime‚1938.267678695 2loss/SRqqt4/assignment_loss‚1.0383172440032165 .loss/FBt3/detection_loss‚3.483721286058426 % _timestamp‚1.7883973136308775e+09 0loss/FRt1/assignment_loss‚3.4028079205370965 2loss/SRqqt1/assignment_loss‚2.0407821533955492 2loss/SRqqt3/assignment_loss‚1.9271604438977583[/Gª_¿Ù /loss/FRt1/detection_loss‚2.1985551334441977 1loss/SRqqt1/detection_loss‚2.0566542778375014 0loss/FBt1/assignment_loss‚0.8342177606920648 /loss/FRt2/detection_loss‚2.9523658983169065 /loss/FRt4/detection_loss‚1.7357939194355692 _runtime‚1938.267678695 2loss/SRqqt4/assignment_loss‚1.0383172440032165 .loss/FBt3/detection_loss‚3.483721286058426 % _timestamp‚1.7883973136308775e+09 0loss/FRt1/assignment_loss‚3.4028079205370965 2loss/SRqqt1/assignment_loss‚2.0407821533955492 2loss/SRqqt3/assignment_loss‚1.9271604438977583 /loss/FRt3/assignment_loss‚4.122945677489042 1loss/SRqqt2/detection_loss‚2.4332172724935743 0loss/SRqqt3/detection_loss‚2.254420119524002 %loss/total_loss‚51.66928065586248 epoch‚1 /loss/FRt4/assignment_loss‚1.949116973649888 /loss/FRt3/detection_loss‚2.7812960231676698 1loss/SRqqt4/detection_loss‚1.1448031397660574 /loss/FBt4/detection_loss‚0.8310364947737638 _step‚194 1loss/FBt4/assignment_loss‚0.28392823956245233 1loss/SRqqt2/assignment_loss‚2.463334016005198 1loss/FBt2/assignment_loss‚0.43586360447525296 .loss/FBt1/detection_loss‚2.389722916212949 .loss/FBt2/detection_loss‚3.356252211332321 trainer/global_step‚2399 /loss/FRt2/assignment_loss‚5.331000135790918 1loss/FBt3/assignment_loss‚0.22196779400110245ÂÔ|È’À‚(:‡ ŽãÔ©Ó¹® gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚117.022' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4044 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0bBjæyÁ‚(:î ŽãÔÔì’²$ disk./.usageGB‚260.7538146972656 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.sda1.out‚0$ memory_percent‚89.84053134918213 disk.dm-1.out‚ 5110.19921875 disk.drbd2266.out‚0 disk.rbd0.out‚0% disk.nvme0n1.out‚18437.2373046875 network.recv‚734081 disk.sda3.in‚ 0.05078125 disk.sda3.out‚ 5110.19921875 disk.nvme0n1.in‚ 334.1484375) disk./.usagePercent‚17.49940729135673 disk.dm-1.in‚ 0.05078125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 8.06494140625! disk.nvme0c0n1.in‚ 334.1484375) proc.memory.availableMB‚ 1664.52734375 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 334.1484375' disk.nvme0c0n1.out‚18437.2373046875* proc.memory.percent‚16.336965560913086 disk.sda.in‚ 0.05078125 disk.drbd2080.in‚0 disk.rbd0.in‚0" disk.dm-0.out‚18437.2373046875 disk.rbd1.in‚0" proc.memory.rssMB‚ 2676.6484375 cpu‚25.23110065256994 proc.cpu.threads‚17 network.sent‚ 4.749417e+06 disk.sda.out‚ 5110.19921875 *Z‚(:’ œŽãÔ‰û´® gpu.0.gpu‚28.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚115.956' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.191200000000002 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸eexÂ(:í œŽãÔøÛ½±% disk./.usageGB‚260.75387954711914 disk.rbd0.out‚0 disk.dm-0.in‚ 334.1484375 disk.sda.in‚ 0.05078125 disk.sda2.out‚0# proc.memory.rssMB‚ 2687.82421875* proc.memory.percent‚16.405177116394043 cpu‚25.262793738818612 network.sent‚ 4.857766e+06 disk.drbd2266.out‚0! disk.nvme0c0n1.in‚ 334.1484375 disk.sda.out‚ 5145.37109375) disk./.usagePercent‚17.49941164348481 disk.rbd1.out‚ 8.21337890625 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0! disk.dm-0.out‚18438.068359375 disk.sda3.out‚ 5145.37109375& disk.nvme0c0n1.out‚18438.068359375 network.recv‚740441 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 334.1484375 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0$ memory_percent‚89.90893363952637 disk.dm-1.out‚ 5145.37109375 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚18438.068359375 disk.dm-1.in‚ 0.05078125 disk.sda1.in‚0 disk.sda3.in‚ 0.05078125( proc.memory.availableMB‚ 1653.3203125 proc.cpu.threads‚17±±úöÄ‚(:… «ŽãÔÈí‘® gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚121.625' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.325 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FñxZvÅ‚(:ë «ŽãԳ쭲 disk.rbd1.out‚ 8.21337890625 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.05078125 disk.drbd2080.out‚0 disk.sda.in‚ 0.05078125 disk.dm-1.out‚ 5196.47265625 network.sent‚ 4.863998e+06 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 334.1484375 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 334.1484375 disk.sda3.in‚ 0.05078125# proc.memory.rssMB‚ 2642.73046875 proc.cpu.threads‚17 disk.drbd1716.out‚0* disk./.usagePercent‚17.499411899492344 disk.sda3.out‚ 5196.47265625 disk.rbd1.in‚0& disk.nvme0c0n1.out‚18438.388671875 disk.sda.out‚ 5196.47265625 disk.drbd2266.out‚0) proc.memory.availableMB‚ 1698.21484375$ disk./.usageGB‚260.7538833618164 disk.drbd2080.in‚0$ disk.nvme0n1.out‚18438.388671875 disk.sda2.out‚0( proc.memory.percent‚16.1299467086792 cpu‚25.29646649029721 network.recv‚745276 disk.sda1.out‚0 disk.rbd0.in‚0! disk.dm-0.out‚18438.388671875! disk.nvme0c0n1.in‚ 334.1484375 disk.drbd2266.in‚0$ memory_percent‚89.63491916656494 disk.sda1.in‚0gË>o‘Æ‚(:† ºŽãÔ膢® gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚143.542' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.7084 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0MhÇßsÇ‚(:è ºŽãÔ»¯†² proc.memory.rssMB‚ 2678.78125 cpu‚25.330907568118526 disk.dm-0.in‚ 334.1484375" disk.dm-0.out‚18438.6865234375 disk.sda1.in‚0 disk.sda.out‚ 5246.078125 disk.drbd2266.in‚0 disk.drbd1716.out‚0 network.sent‚ 4.869496e+06% disk./.usageGB‚260.75386810302734 disk.drbd2265.out‚0 disk.sda3.out‚ 5246.078125 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 8.24560546875' disk.nvme0c0n1.out‚18438.6865234375 disk.dm-1.out‚ 5246.078125 disk.sda1.out‚0 disk.drbd2080.out‚0% disk.nvme0n1.out‚18438.6865234375 disk.sda2.out‚0) proc.memory.percent‚16.34998321533203 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda.in‚ 0.05078125 disk.dm-1.in‚ 0.05078125 disk.drbd2266.out‚0 disk.nvme0n1.in‚ 334.1484375 network.recv‚749013* disk./.usagePercent‚17.499410875462207 disk.drbd2265.in‚0 disk.sda3.in‚ 0.05078125 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 334.1484375 disk.sda2.in‚0 disk.drbd1716.in‚0$ memory_percent‚89.85540866851807) proc.memory.availableMB‚ 1662.08984375í³©ð2È- _wandbruntime‚2003  _runtime‚2003ŒKÙ.*ɨT YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 3409137449984 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚416 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚416 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 3468692933632 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚80683775 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 145310208 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚281 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚72147210196992 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚217 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24126 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚31830632 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24126 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚64771409045504 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚48853145 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚217 ]JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚71499422069248 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 127307264 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚80683775 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 265683968 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚68170848550799 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚80699752 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚13807830 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 109670912 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚13808111 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 3468802604544 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 3399439505295 _step‚195 VCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚71499276759040 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚68090139309056 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚64771392006144 trainer/global_step‚2417 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15977 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚26 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚982 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 145310208 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚31830630 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 406789961 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 417333248 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚217 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2460 _runtime‚2003.715545178 YFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚71499276759040 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 410711040 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚48853145 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 127307264 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚31830632 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15975 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 4757245249 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚199 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24399 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚416 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚18581169 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 3399315579711 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15975 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚32388997 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 133611008 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚80699752 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚32389280 ZGDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚71499422069248 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚283 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚199 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15977 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 4912019968 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚68090157312000 YFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚68170707585855 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 4912019968 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚48869120 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚72147234137088‰PÚbg Ê­T UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚48853145 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 127307264 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚31830632 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15975 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 4757245249 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚199 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24399 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚416 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚18581169 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 3399315579711 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15975 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚32388997 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 133611008 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 NADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚80699752 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚32389280 ZGDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚71499422069248 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚283 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚199 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15977 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 4912019968 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.alürìÌlocated‚68090157312000 YFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚68170707585855 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 4912019968 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚48869120 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚72147234137088 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 3409137449984 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚416 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚416 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 3468692933632 J=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚80683775 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 145310208 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚281 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚72147210196992 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚217 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24126 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚31830632 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24126 dQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚64771409045504 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚48853145 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚217 ]JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚71499422069248 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 127307264 NADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚80683775 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 265683968 ]JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚68170848550799 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚80699752 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚13807830 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 109670912 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚13808111 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 3468802604544 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 3399439505295 _step‚195 VCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚71499276759040 ]JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚68090139309056 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚64771392006144 trainer/global_step‚2417 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15977 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚26 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚982 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 145310208 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚31830630 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 406789961 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 417333248 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚217 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2460 _runtime‚2003.715545178 YFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚71499276759040 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 410711040é÷Å2Ë- _wandbruntime‚2004  _runtime‚2004s¹ÕZ8)̲R WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚31841344 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚31841342 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 3469151740416 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 123925584 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 4624220160 ]KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚ 3409486585856 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 410711040 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2460 `MDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚75635989443072 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 109670912 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 4757245249 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15977 H;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚80701883 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚31841344 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚982 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚64789293559808 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚199 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 410711040 TADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚71518491080704 \IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚75635855832064 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ]KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚ 3399663186047 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 5041553408 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 5041553408 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚416 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚217 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚68109004494848 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 % _timestamp‚1.7883973800217032e+09 WDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚71518491080704 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚72166837702656 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 265683968 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚72166813762560 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 3399787111631 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24399 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 127307264 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚68109022497792 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 % _timestamp‚1.7883973800217032e+09 WDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚71518491080704 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚72166837702656 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 265683968 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚72166813762560 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 aODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚ 3399787111631 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24399 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 127307264 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚68109022497792 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 B&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_1_of_2‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.06413893401622772 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.8720340132713318 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 )loss/val_total_loss‚50.95982216129478 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2633184492588043 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.3947368562221527 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.051257167011499405 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4012451469898224 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.2994047701358795 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.013044847175478935 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4936794936656952 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14785386621952057 ,jet/accuracy_3_of_5‚0.028603430837392807 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4696842432022095 jet/accuracy_7_of_7‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.46539580821990967 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02348875254392624 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19130964577198029 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.0957622081041336 *jet/accuracy_2_of_6‚0.3461538553237915 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.09000549465417862 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9713776707649231 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9425286650657654 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7045454382896423 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.39054393768310547 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2722362279891968 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9464285969734192 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9848198294639587 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.0833333358168602 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9848105907440186 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.29396191239356995 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9864341616630554 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5420316457748413 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9829689264297485 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8231077194213867 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.09152045845985413 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3987329304218292 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5061728358268738 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.25838178396224976 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08028470724821091 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.001792114693671465 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4676848351955414 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.07228066772222519 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4804597795009613 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 +jet/accuracy_3_of_6‚0.07692307978868484 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4936794936656952 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9814789891242981 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.32499998807907104 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_7_of_7‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4324638843536377 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 jet/accuracy_5_of_5‚0 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_4‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166666865348816 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.28908586502075195 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5009192824363708 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9852567911148071 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.28125 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0026615362148731947 jet/accuracy_5_of_6‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.49908119440078735 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2657407820224762 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.31112173199653625 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09281811118125916 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8368779420852661 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9779592752456665 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.011174988932907581 *jet/accuracy_1_of_6‚0.9230769276618958 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 _step‚197 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.10185541957616806 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.11486486345529556 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.20462962985038757 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4112200140953064 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.08641643077135086 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 *jet/accuracy_1_of_1‚0.4613434672355652 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_7‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.011712750419974327 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 jet/accuracy_1_of_7‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2220286726951599 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_2‚0.7130411267280579 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_7‚0.5 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.44050294160842896 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.44493937492370605 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1346491128206253 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9678160548210144 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.09927476942539215 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 trainer/global_step‚2417 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.010057471692562103 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.003710574936121702 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,particle/sensitivity‚0.17670698463916779 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9910743236541748 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09775463491678238 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13044562935829163 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3744107186794281 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.28368034958839417 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5323988795280457 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.10030938684940338 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.08222221583127975 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9924528002738953 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.36240309476852417 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5267857313156128 *jet/accuracy_1_of_3‚0.8284574747085571 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4992462992668152 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 particle/accuracy_4_of_5‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3541666567325592 *"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.09560820460319519 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.31481480598449707 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.8999999761581421 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.44117647409439087 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9846027493476868 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5522372722625732 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 6 Purity/1FRt3SRqqt*FBt/FBt_purity‚0.970588207244873 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.47493913769721985 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.3995966613292694 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1577380895614624 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09577123820781708 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_4‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4113289713859558 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4735229015350342 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5114942789077759 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2613901197910309 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6327208876609802 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09462405741214752 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4572327136993408 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_3‚1 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.47999998927116394 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14320002496242523 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4685553014278412 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9259259104728699 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2081308364868164 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15648554265499115 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_7‚1 _runtime‚2017.341888436 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4414004385471344 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.010501675307750702 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_5‚1 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.995573878288269 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.4375 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.2858738601207733 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.2697530686855316 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4056374430656433 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3086419999599457 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5104469656944275 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9954687356948853 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4929194152355194 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_7‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9727749228477478 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9300000071525574 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4166666865348816 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4463939666748047 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_7‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9944243431091309 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.3769382834434509 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9929491281509399 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.11611669510602951 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1535494029521942 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15356670320034027 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_7‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3284052014350891 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0016544089885428548 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_1_of_4‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.09358002245426178 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.08063748478889465 particle/accuracy_2_of_5‚1 'particle/f_score‚0.2526520788669586 particle/accuracy_2_of_7‚1 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.25 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.321800172328949 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.014098231680691242 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.48237037658691406 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9590260982513428 jet/accuracy_5_of_7‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11815139651298523 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9746935963630676 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10643431544303894 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.009523809887468815 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 particle/accuracy_5_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9745181202888489 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9907218813896179 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 $ _timestamp‚1.788397392702667e+09 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10283686965703964 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9820483922958374 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16448982059955597 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4884394109249115 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9614124298095703 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.2724982500076294 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.15311133861541748 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.13131311535835266 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.09848485141992569 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4613661766052246 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9472222328186035 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_6‚0 epoch‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1071428582072258 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.38995349407196045 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4922221899032593 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.48245614767074585 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.021967250853776932 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.48880988359451294 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6300339698791504 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3402777910232544 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9517537951469421 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.44983533024787903 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_6_of_7‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5209987759590149 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4269345998764038 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.00515872985124588 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16149845719337463 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.11508847773075104 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.987072765827179 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9783625602722168 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09502707421779633 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4694386422634125 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28565749526023865 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.12155172228813171 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 *jet/accuracy_1_of_5‚0.9276625514030457 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.05892137065529823 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.47862622141838074 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7045454382896423 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.08127571642398834 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.04124149680137634 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4381992220878601 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.43368056416511536 8 Purity/2FRt0SRqqt£ýáI 2FBt/FRt_purity‚0.39254388213157654 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 )jet/accuracy_2_of_5‚0.385010302066803 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.416301429271698 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4166666865348816 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.14439241588115692 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_3_of_6‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04020969569683075 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5199999809265137 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4559664726257324 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9891326427459717 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.44668203592300415 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.07737015932798386 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17509444057941437 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5269774198532104 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.954960286617279 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.033647798001766205 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4860806167125702 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.09827060997486115 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4803571403026581 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3437500298023224 & Purity/2FRt3SRqqt1FBt/FRt_purity‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3561509847640991 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094ßÙÈCésÒ“¦ 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 *jet/accuracy_1_of_1‚0.4613434672355652 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_7‚0 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.011712750419974327 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 jet/accuracy_1_of_7‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 & Purity/4FRt1SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2220286726951599 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_2‚0.7130411267280579 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_6_of_7‚0.5 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.44050294160842896 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 :"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.44493937492370605 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1346491128206253 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9678160548210144 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.09927476942539215 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2222222238779068 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 trainer/global_step‚2417 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.010057471692562103 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.003710574936121702 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,particle/sensitivity‚0.17670698463916779 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9910743236541748 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.09775463491678238 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.13044562935829163 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3744107186794281 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9642857313156128 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.28368034958839417 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5323988795280457 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.10030938684940338 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.08222221583127975 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9924528002738953 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.36240309476852417 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5267857313156128 *jet/accuracy_1_of_3‚0.8284574747085571 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.4992462992668152 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 particle/accuracy_4_of_5‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3541666567325592 *"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.5 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.09560820460319519 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.31481480598449707 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.8999999761581421 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.44117647409439087 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9846027493476868 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5522372722625732 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 6 Purity/1FRt3SRqqt*FBt/FBt_purity‚0.970588207244873 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/event_purity‚0.1111111119389534 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.47493913769721985 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.3995966613292694 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.1577380895614624 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.10000000149011612 9"Purity/0FRt1SRqqt3FBt/event_purity‚0.1428571492433548 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.09577123820781708 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_4_of_4‚1 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4113289713859558 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.4735229015350342 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5114942789077759 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2613901197910309 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6327208876609802 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.09462405741214752 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4572327136993408 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_3‚1 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.47999998927116394 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.095238097012043 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.14320002496242523 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4685553014278412 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9259259104728699 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2081308364868164 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.15648554265499115 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_7‚1 _runtime‚2017.341888436 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4414004385471344 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.010501675307750702 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_5‚1 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.995573878288269 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.4375 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.2858738601207733 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.2697530686855316 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4056374430656433 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3086419999599457 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.10000000149011612 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5104469656944275 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9954687356948853 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.4929194152355194 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_4_of_7‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9727749228477478 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9300000071525574 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4166666865348816 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4463939666748047 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_7‚1 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9944243431091309 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.3769382834434509 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9929491281509399 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.11611669510602951 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1535494029521942 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.15356670320034027 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_7‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3284052014350891 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 <"Purity/2FRt*SRqqt*FBt/event_purity‚0.0016544089885428548 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 particle/accuracy_1_of_4‚1 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.09358002245426178 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.08063748478889465 particle/accuracy_2_of_5‚1 'particle/f_score‚0.2526520788669586 particle/accuracy_2_of_7‚1 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.25 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.321800172328949 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.014098231680691242 :"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.48237037658691406 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9590260982513428 jet/accuracy_5_of_7‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.11815139651298523 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9746935963630676 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.10643431544303894 ;"Purity/2FRt0SRqqt1FBt/event_purity‚0.009523809887468815 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 particle/accuracy_5_of_5‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9745181202888489 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9907218813896179 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 $ _timestamp‚1.788397392702667e+09 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/4FRt1SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10283686965703964 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9820483922958374 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.16448982059955597 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.4884394109249115 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ( Purity/*FRt3SRqqt2FBt/FBt_purity‚0.5 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9614124298095703 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.2724982500076294 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.15311133861541748 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.13131311535835266 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.09848485141992569 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4613661766052246 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9472222328186035 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 jet/accuracy_4_of_6‚0 epoch‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1071428582072258 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.38995349407196045 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚0 & Purity/4FRt*SRqqt*FBt/FRt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.4922221899032593 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.48245614767074585 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.021967250853776932 :"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.48880988359451294 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6300339698791504 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3402777910232544 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9517537951469421 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.44983533024787903 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 jet/accuracy_6_of_7‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5209987759590149 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4269345998764038 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.00515872985124588 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.16149845719337463 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.11508847773075104 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.987072765827179 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9783625602722168 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.09502707421779633 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4694386422634125 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28565749526023865 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.12155172228813171 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 *jet/accuracy_1_of_5‚0.9276625514030457 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.05892137065529823 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.47862622141838074 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7045454382896423 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.08127571642398834 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.04124149680137634 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4381992220878601 :"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.43368056416511536 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.39254388213157654 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 )jet/accuracy_2_of_5‚0.385010302066803 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 8"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.416301429271698 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4166666865348816 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.14439241588115692 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 particle/accuracy_3_of_6‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.04020969569683075 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5199999809265137 ( Purity/1FRt3SRqqt2FBt/FBt_purity‚0.5 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4559664726257324 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9891326427459717 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.44668203592300415 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4642857015132904 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.07737015932798386 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.17509444057941437 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5269774198532104 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.954960286617279 ("Purity/2FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.033647798001766205 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.4860806167125702 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.09827060997486115 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.4803571403026581 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3437500298023224 & Purity/2FRt3SRqqt1FBt/FRt_purity‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.3561509847640991 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 +"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.75 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_3_of_4‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8859648704528809 jet/accuracy_4_of_7‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.04657374322414398 :"Purity/1FRt1SRqqt0FBt/event_purity‚0.01006336696445942 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.23749999701976776 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3564564287662506 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 6 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.375619113445282 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.990590512752533 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05505017936229706 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 particle/accuracy_4_of_6‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.36917778849601746 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.43046003580093384 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.44178393483161926 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.4652823507785797 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.08641643077135086 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.39427462220191956 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.14773619174957275 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3645833432674408 :"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.43518519401550293 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9101656079292297 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2083333283662796 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.0416666679084301 particle/accuracy_2_of_6‚1 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.19628532230854034 jet/accuracy_6_of_6‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.37350279092788696 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.01315789483487606 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4832382798194885 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.19688300788402557 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.09906081855297089 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.39471617341041565 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.1755947470664978 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1315789520740509 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.0416666679084301 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3888888955116272 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_1_of_2‚1 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.06413893401622772 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_4‚0.8720340132713318 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 )loss/val_total_loss‚50.95982216129478 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.2633184492588043 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.3947368562221527 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ;"Purity/0FRt2SRqqt*FBt/event_purity‚0.051257167011499405 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4012451469898224 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.2994047701358795 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.013044847175478935 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4936794936656952 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.14785386621952057 ,jet/accuracy_3_of_5‚0.028603430837392807 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4696842432022095 jet/accuracy_7_of_7‚0 :"Purity/*FixN1Rt*SRqqt*FBt/SRqqt_purity‚0.46539580821990967 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.02348875254392624 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19130964577198029 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.0957622081041336 *jet/accuracy_2_of_6‚0.3461538553237915 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.09000549465417862 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9713776707649231 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9425286650657654 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7045454382896423 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ("Purity/2FRt0SRqqt0FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.39054393768310547 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.2722362279891968 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9464285969734192 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9848198294639587 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.0833333358168602 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9848105907440186 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.29396191239356995 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9864341616630554 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5420316457748413 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9829689264297485 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8231077194213867 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.09152045845985413 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3987329304218292 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.007751938421279192 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5061728358268738 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.25838178396224976 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.08028470724821091 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,jet/accuracy_4_of_5‚0.001792114693671465 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4676848351955414 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.07228066772222519 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4804597795009613 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 +jet/accuracy_3_of_6‚0.07692307978868484 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4936794936656952 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9814789891242981 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.32499998807907104 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_7_of_7‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4324638843536377 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 jet/accuracy_5_of_5‚0 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_2‚1 particle/accuracy_1_of_3‚1 particle/accuracy_2_of_4‚1 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166666865348816 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.28908586502075195 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5009192824363708 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9852567911148071 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.28125 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0026615362148731947 jet/accuracy_5_of_6‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚0 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.49908119440078735 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.2657407820224762 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 particle/accuracy_1_of_6‚1 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.31112173199653625 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.09281811118125916 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 (particle/accuracy‚0.8368779420852661 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9779592752456665 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.011174988932907581 *jet/accuracy_1_of_6‚0.9230769276618958 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.625 & Purity/4FRt*SRqqt0FBt/FRt_purity‚0 _step‚197 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.10185541957616806 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.11486486345529556 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.20462962985038757 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4112200140953064 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.08641643077135086 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0ÅÃl.O’Ó‚(:‡ ØŽãÔ¾±¨® gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚117.022' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4044 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]¶±{Ô‚(:ð ØŽãÔ¼Œ›± disk.drbd2265.out‚0 disk.sda3.in‚ 0.05078125! disk.nvme0c0n1.in‚ 334.1484375 disk.nvme0n1.in‚ 334.1484375 disk.dm-1.in‚ 0.05078125* proc.memory.percent‚16.441845893859863 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0& disk.nvme0n1.out‚18620.63232421875 proc.cpu.threads‚17 disk.sda.out‚ 5328.24609375 disk.drbd2265.in‚0# disk.dm-0.out‚18620.63232421875 cpu‚25.381087799456406 disk.drbd2080.out‚0 disk.dm-0.in‚ 334.1484375 disk.sda3.out‚ 5328.24609375# proc.memory.rssMB‚ 2693.83203125% disk./.usageGB‚252.21501541137695 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 8.30859375( disk.nvme0c0n1.out‚18620.63232421875 network.recv‚759779 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 5328.24609375# memory_percent‚90.6989336013794) proc.memory.availableMB‚ 1523.88671875 network.sent‚ 4.983425e+06) disk./.usagePercent‚16.92636130675093 disk.sda.in‚ 0.05078125 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0„ËÚÕ‚(:’ çŽãÔªœ® gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚133.197' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.639400000000002 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)8ý|{Ö‚(:ð çŽãÔ¼ÅDZ network.recv‚766271 disk.rbd0.in‚0# disk.dm-0.out‚18632.99951171875 disk.rbd1.in‚0 disk.dm-1.in‚ 0.05078125( proc.memory.availableMB‚ 1523.8828125* proc.memory.percent‚16.441845893859863* disk./.usagePercent‚16.926328025771483 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 8.30859375& disk.nvme0n1.out‚18632.99951171875 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 334.1484375 disk.sda2.in‚0 disk.dm-0.in‚ 334.1484375( disk.nvme0c0n1.out‚18632.99951171875 disk.sda.in‚ 0.05078125 disk.dm-1.out‚ 5367.81640625% disk./.usageGB‚252.21451950073242 disk.rbd0.out‚0 disk.sda3.in‚ 0.05078125 proc.cpu.threads‚17 disk.sda2.out‚0 network.sent‚ 5.107723e+06 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.sda3.out‚ 5367.81640625 disk.sda.out‚ 5367.81640625 disk.drbd1716.in‚0 disk.drbd1716.out‚0! disk.nvme0c0n1.in‚ 334.1484375# memory_percent‚90.6989574432373# proc.memory.rssMB‚ 2693.83203125 cpu‚25.378794538641646 disk.sda1.in‚0Ê¥2×- _wandbruntime‚2043  _runtime‚2043˜¤*ùÈØ _runtime‚2043.074313878 _step‚198 ' lr-AdamW/pg1‚0.00020817087188822813 ' lr-AdamW/pg2‚0.00020817087188822813 trainer/global_step‚2449 $ _timestamp‚1.788397418437433e+09NPÍÙÇ ' lr-AdamW/pg1‚0.00020817087188822813 ' lr-AdamW/pg2‚0.00020817087188822813 trainer/global_step‚2449 $ _timestamp‚1.788397418437433e+09 _runtime‚2043.074313878 _step‚198ÆHöBÛ2Ú- _wandbruntime‚2043  _runtime‚20433çàLÇ'ÛÁO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18900854 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚76835463885312 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14041978 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3447217215265 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚65826269449216 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49571439 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876544 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚72656974363136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32373029 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32942834 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7883974184381394e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73317985449984 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚72656787360768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32373027 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚72656974363136 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14041476 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81944466 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚502 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚65826252409856 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚69199620219904 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3517719251968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚72656787360768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49591476 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚69199620219904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49571439 _runtime‚2043.075290311 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚73317961509888 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _step‚199 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32942330 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3457354143232 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3457185143808 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3517502375424 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3457185143808 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037©ƒ”ÉÌ'ÜÆO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚49591476 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚32373027 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3447382336665 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚18900856 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚32373029 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚69199620219904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚49571439 _runtime‚2043.075290311 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚73317961509888 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _step‚199 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚32942330 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3457354143232 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3457185143808 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3517502375424 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3457185143808 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚18900854 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚76835463885312 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14041978 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3447217215265 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚65826269449216 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚49571439 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876544 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚72656974363136 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚32373029 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚32942834 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7883974184381394e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚73317985449984 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚72656787360768 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚32373027 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚72656974363136 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14041476 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚81944466 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚502 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚65826252409856 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚69199620219904 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚504 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3517719251968 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚72656787360768 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚49591476 IDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _step‚200 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81976912 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32385686 REDeviceStatsMonitor.on_train_batch_end/allocatió×bD>$on.large_pool.allocated‚32385688 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49611289 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81996977 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3448606379809 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32385688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14047248 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76865804920832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚2449 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚72685636879360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3458747294720 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚69300984132969 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3458578281984 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32955481 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚2043.771064375 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49591226 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18908233 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18908231 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49611289 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚69226871581696 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚72685449863680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚69300801972001 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚69226889584640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚69226871581696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚73346898398208 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚72685636879360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49591226 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214766080 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3448771501417 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3458747294720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81996977 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32954997 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 # _timestamp‚1.78839741913364e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚65852195592192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32385686 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚484mª¼–Ô&ßÎM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3458578281984 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚32955481 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚2043.771064375 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚49591226 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚18908233 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚18908231 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚49611289 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚69226871581696 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚72685449863680 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚69300801972001 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚69226889584640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚69226871581696 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚73346898398208 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚72685636879360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚49591226 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214766080 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3448771501417 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3458747294720 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚81996977 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ODeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚32954997 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 # _timestamp‚1.78839741913364e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚65852195592192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚32385686 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚484 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _step‚200 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚81976912 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚32385686 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚32385688 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚49611289 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚81996977 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3448606379809 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚32385688 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚482 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14047248 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚76865804920832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚2449 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚72685636879360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3458747294720 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚69300984132969 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224È÷¡ÕÍ2à- _wandbruntime‚2043  _runtime‚2043|/bŠTáÎ 1loss/SRqqt3/assignment_loss‚1.519311934709549 /loss/FRt4/detection_loss‚1.7387465146325884 0loss/SRqqt3/detection_loss‚2.331725997038377 %loss/total_loss‚50.28128896604261 _runtime‚2043.772215859 2loss/SRqqt1/assignment_loss‚1.8731454195929509 0loss/FBt2/assignment_loss‚0.3536831904432347 -loss/FRt2/detection_loss‚2.87286171572549 1loss/SRqqt4/assignment_loss‚1.132212654288326 0loss/FBt1/assignment_loss‚0.5812125835906375 0loss/FBt4/assignment_loss‚0.3984864421429735 1loss/SRqqt2/detection_loss‚2.4102250981975244 epoch‚2 $ _timestamp‚1.788397419135375e+09 /loss/FRt3/assignment_loss‚3.319156050682068 /loss/FRt4/assignment_loss‚2.166299129289294 trainer/global_step‚2449 1loss/SRqqt4/detection_loss‚1.2427542927009718 /loss/FBt1/detection_loss‚2.7266971197995273 /loss/FBt2/detection_loss‚2.9955777302384377 /loss/FRt1/assignment_loss‚4.087205345630646 /loss/FRt2/assignment_loss‚4.366852082524981 0loss/FBt3/assignment_loss‚0.7973038554191589 /loss/FRt1/detection_loss‚2.2043896770477294 /loss/FBt3/detection_loss‚3.4250909984111786 /loss/FBt4/detection_loss‚0.8249101656827771 2loss/SRqqt2/assignment_loss‚2.4618213523078607 1loss/SRqqt1/detection_loss‚1.9625888420086282 _step‚201 .loss/FRt3/detection_loss‚2.4890307739377028‘ˆYâÓ 1loss/SRqqt3/assignment_loss‚1.519311934709549 /loss/FRt4/detection_loss‚1.7387465146325884 0loss/SRqqt3/detection_loss‚2.331725997038377 %loss/total_loss‚50.28128896604261 _runtime‚2043.772215859 2loss/SRqqt1/assignment_loss‚1.8731454195929509 0loss/FBt2/assignment_loss‚0.3536831904432347 -loss/FRt2/detection_loss‚2.87286171572549 1loss/SRqqt4/assignment_loss‚1.132212654288326 0loss/FBt1/assignment_loss‚0.5812125835906375 0loss/FBt4/assignment_loss‚0.3984864421429735 1loss/SRqqt2/detection_loss‚2.4102250981975244 epoch‚2 $ _timestamp‚1.788397419135375e+09 /loss/FRt3/assignment_loss‚3.319156050682068 /loss/FRt4/assignment_loss‚2.166299129289294 trainer/global_step‚2449 1loss/SRqqt4/detection_loss‚1.2427542927009718 /loss/FBt1/detection_loss‚2.7266971197995273 /loss/FBt2/detection_loss‚2.9955777302384377 /loss/FRt1/assignment_loss‚4.087205345630646 /loss/FRt2/assignment_loss‚4.366852082524981 0loss/FBt3/assignment_loss‚0.7973038554191589 /loss/FRt1/detection_loss‚2.2043896770477294 /loss/FBt3/detection_loss‚3.4250909984111786 /loss/FBt4/detection_loss‚0.8249101656827771 2loss/SRqqt2/assignment_loss‚2.4618213523078607 1loss/SRqqt1/detection_loss‚1.9625888420086282 _step‚201 .loss/FRt3/detection_loss‚2.489030773937702ÉÄ™¶±‘ã‚(:† öŽãÔ“ù£¯ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚113.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7068 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0š8Orä‚(:ç öŽãÔ£·Á°! disk.nvme0c0n1.in‚ 334.1484375 proc.cpu.threads‚17 network.sent‚ 5.113068e+06 disk.sda.out‚ 5410.0078125 network.recv‚769301 disk.dm-0.out‚18744.24609375 disk.nvme0n1.in‚ 334.1484375 disk.sda.in‚ 0.05078125 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.dm-0.in‚ 334.1484375 disk.sda3.out‚ 5410.0078125% disk./.usageGB‚252.21451950073242 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.05078125 disk.rbd1.in‚0 disk.drbd2266.out‚0* disk./.usagePercent‚16.926328025771483 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.rbd1.out‚ 8.37158203125 disk.sda2.in‚0 disk.dm-1.out‚ 5410.0078125# proc.memory.rssMB‚ 2693.83203125* proc.memory.percent‚16.441845893859863 disk.rbd0.out‚0 disk.sda1.in‚0% disk.nvme0c0n1.out‚18744.24609375# disk.nvme0n1.out‚18744.24609375 disk.dm-1.in‚ 0.05078125$ memory_percent‚90.19169807434082( proc.memory.availableMB‚ 1606.9921875 cpu‚25.376329889397240)bg’å‚(:‡ …ãÔö²£® gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.931' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7862 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íؾúwæ‚(:ì …ãÔ¤Çò° disk.rbd1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.05078125 cpu‚25.373983689298385% disk./.usageGB‚252.21451950073242% disk.nvme0c0n1.out‚18744.49609375# disk.nvme0n1.out‚18744.49609375 disk.drbd2265.in‚0 proc.cpu.threads‚17* disk./.usagePercent‚16.926328025771483 disk.rbd0.out‚0 disk.sda3.in‚ 0.05078125 disk.sda3.out‚ 5443.95703125 disk.nvme0n1.in‚ 334.1484375 disk.rbd0.in‚0 disk.dm-0.out‚18744.49609375 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 8.37158203125 disk.drbd2266.in‚0 disk.drbd2265.out‚0 network.sent‚ 5.221282e+06 network.recv‚775230 disk.drbd2080.out‚0 disk.drbd1716.out‚0! disk.nvme0c0n1.in‚ 334.1484375 disk.sda2.in‚0* proc.memory.percent‚16.441845893859863 disk.drbd2080.in‚0 disk.dm-0.in‚ 334.1484375 disk.drbd1716.in‚0 disk.dm-1.out‚ 5443.95703125 disk.drbd2266.out‚0$ memory_percent‚90.67995548248291 disk.sda.in‚ 0.05078125 disk.sda.out‚ 5443.95703125) proc.memory.availableMB‚ 1526.99609375# proc.memory.rssMB‚ 2693.83203125½>¡z2ç- _wandbruntime‚2080  _runtime‚2080³Â¡ÝÉèà % _timestamp‚1.7883974556152148e+09 _runtime‚2080.252113122 _step‚202 ' lr-AdamW/pg1‚0.00020761662728484482 ' lr-AdamW/pg2‚0.00020761662728484482 trainer/global_step‚2499jW«ôÎéÈ ' lr-AdamW/pg1‚0.00020761662728484482 ' lr-AdamW/pg2‚0.00020761662728484482 trainer/global_step‚2499 % _timestamp‚1.7883974556152148e+09 _runtime‚2080.252113122 _step‚202Ê?Êí2ê- _wandbruntime‚2080  _runtime‚2080e‡°ý¦ëÁO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚74763632861184 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚50584178 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚67123411526656 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19269704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14313255 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚50564141 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234525184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33583451 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚78352095650816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚70563070453760 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83590157 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19269706 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚70563070453760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3526856466432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33005979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚74763608921088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚70640100560329 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 _runtime‚2080.253076801 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74090113922560 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚490 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210585088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33582959 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3588486729728 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚70640282721089 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3516854155073 % _timestamp‚1.7883974556159728e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚74089926920192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33005977 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74089926920192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor2! .on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3527025465856 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚74090113922560 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50584178 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚203 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚74089926920192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3527025465856 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚74090113922560 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚50584178 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚203 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234525184 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚33583451 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚78352095650816 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚70563070453760 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚83590157 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19269706 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚70563070453760 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3526856466432 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33005979 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚74763608921088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚70640100560329 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 _runtime‚2080.253076801 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚74090113922560 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚490 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210585088 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚33582959 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3588486729728 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚70640282721089 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3516854155073 % _timestamp‚1.7883974556159728e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚74089926920192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚33005977 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 trainer/global_step‚2499 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83622629 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3590114516480 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3518243319825 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3528249604608 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3528249604608 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74792521869312 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33595576 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33018636 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33596074 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83622629 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3528418617344 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33018638 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚74118589423104 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚70590339818496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19277083 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚70667432907209 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚74118776438784 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _step‚204 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3589908139008 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50583928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19277081 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚498 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚78382660325888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50603991 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚78382430008320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206377472 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14318991 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3518078198217 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33018636 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚67149371748352 _runtime‚2080.954308985 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83602564 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚70590339818496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70667615068177 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50603991 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ODeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33018636 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚67149371748352 _runtime‚2080.954308985 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚83602564 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚70590339818496 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚70667615068177 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚50603991 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 trainer/global_step‚2499 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚83622629 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3590114516480 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3518243319825 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3528249604608 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3528249604608 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚74792521869312 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚33595576 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33018636 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚33596074 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚83622629 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3528418617344 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33018638 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚74118589423104 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚70590339818496 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19277083 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 265683968 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚70667432907209 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚74118776438784 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _step‚204 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3589908139008 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚50583928 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚19277081 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚498 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [HDevicJ<““éeStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚78382660325888 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚50603991 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚78382430008320 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206377472Ì;#tø2ð- _wandbruntime‚2080  _runtime‚2080•­®hTñÎ 0loss/FRt3/assignment_loss‚3.6771941035985947 /loss/FRt4/assignment_loss‚2.280137261375785 epoch‚2 _step‚205 /loss/FRt1/assignment_loss‚4.181507334417226 /loss/FRt2/detection_loss‚3.4060781371025812 0loss/FBt4/assignment_loss‚0.3314784228137428 2loss/SRqqt1/assignment_loss‚1.8115561939045495 1loss/SRqqt2/assignment_loss‚2.215303312987089 +loss/FRt3/detection_loss‚2.308942489326 1loss/SRqqt2/detection_loss‚2.1608788310126825 .loss/FBt3/detection_loss‚3.182302164179938 &loss/total_loss‚49.857799325873636 % _timestamp‚1.7883974563189387e+09 _runtime‚2080.955702415 2loss/SRqqt3/assignment_loss‚1.8181874890412604 2loss/SRqqt4/assignment_loss‚1.4142462921639283 0loss/FBt3/assignment_loss‚0.6651915013790131 /loss/FRt4/detection_loss‚1.8236532527953386 .loss/FBt2/detection_loss‚3.644676461815834 /loss/FBt1/assignment_loss‚0.480798667685389 /loss/FBt1/detection_loss‚2.0637316440834717 trainer/global_step‚2499 .loss/FRt1/detection_loss‚2.380575195867188 /loss/FRt2/assignment_loss‚3.596619867143177 0loss/FBt2/assignment_loss‚0.4489191071647838 1loss/SRqqt1/detection_loss‚2.1143823283560135 1loss/SRqqt3/detection_loss‚1.9229415442262376 1loss/SRqqt4/detection_loss‚1.2060124362922378 /loss/FBt4/detection_loss‚0.7224852871415767¯ñ3PYòÓ /loss/FBt1/detection_loss‚2.0637316440834717 trainer/global_step‚2499 .loss/FRt1/detection_loss‚2.380575195867188 /loss/FRt2/assignment_loss‚3.596619867143177 0loss/FBt2/assignment_loss‚0.4489191071647838 1loss/SRqqt1/detection_loss‚2.1143823283560135 1loss/SRqqt3/detection_loss‚1.9229415442262376 1loss/SRqqt4/detection_loss‚1.2060124362922378 /loss/FBt4/detection_loss‚0.7224852871415767 0loss/FRt3/assignment_loss‚3.6771941035985947 /loss/FRt4/assignment_loss‚2.280137261375785 epoch‚2 _step‚205 /loss/FRt1/assignment_loss‚4.181507334417226 /loss/FRt2/detection_loss‚3.4060781371025812 0loss/FBt4/assignment_loss‚0.3314784228137428 2loss/SRqqt1/assignment_loss‚1.8115561939045495 1loss/SRqqt2/assignment_loss‚2.215303312987089 +loss/FRt3/detection_loss‚2.308942489326 1loss/SRqqt2/detection_loss‚2.1608788310126825 .loss/FBt3/detection_loss‚3.182302164179938 &loss/total_loss‚49.857799325873636 % _timestamp‚1.7883974563189387e+09 _runtime‚2080.955702415 2loss/SRqqt3/assignment_loss‚1.8181874890412604 2loss/SRqqt4/assignment_loss‚1.4142462921639283 0loss/FBt3/assignment_loss‚0.6651915013790131 /loss/FRt4/detection_loss‚1.8236532527953386 .loss/FBt2/detection_loss‚3.644676461815834 /loss/FBt1/assignment_loss‚0.480798667685389Ír.8)’ó‚(:‡ ”ãÔÖª® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚119.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9466 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘úÛ–}ô‚(:ò ”ãÔ´“â° disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.05078125# proc.memory.rssMB‚ 2693.83203125 disk.drbd2266.out‚0 disk.drbd2080.in‚0# disk.dm-0.out‚18744.90869140625 disk.sda2.in‚0 disk.drbd1716.in‚0* disk./.usagePercent‚16.926362074773536 disk.nvme0n1.in‚ 334.1484375 cpu‚25.371683806047198 network.sent‚ 5.227247e+06% disk./.usageGB‚252.21502685546875 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 8.37158203125! disk.nvme0c0n1.in‚ 334.1484375* proc.memory.percent‚16.441845893859863 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚91.29047393798828 network.recv‚779798 disk.dm-0.in‚ 334.1484375 disk.sda1.in‚0 disk.sda.in‚ 0.05078125 disk.dm-1.out‚ 5470.80078125& proc.memory.availableMB‚ 1426.96875 disk.drbd2265.in‚0 disk.drbd2265.out‚0& disk.nvme0n1.out‚18744.90869140625 disk.sda3.in‚ 0.05078125 disk.sda3.out‚ 5470.80078125( disk.nvme0c0n1.out‚18744.90869140625 disk.sda.out‚ 5470.80078125 ƒ³’õ‚(:‡ £ãÔÕ‰Ò® gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚129.859' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.9718 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EÊ{ö‚(:ð £ãÔˆòŸ±! disk.nvme0c0n1.in‚ 334.1484375 disk.sda.out‚ 5497.45703125 disk.drbd1716.in‚0# proc.memory.rssMB‚ 2693.83203125% disk./.usageGB‚252.21469497680664 disk.drbd2266.in‚0 disk.sda1.in‚0& disk.nvme0n1.out‚18745.56787109375 disk.sda2.out‚0 network.recv‚786282 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 334.1484375 disk.sda3.out‚ 5497.45703125* proc.memory.percent‚16.441845893859863* disk./.usagePercent‚16.926339802118054 disk.drbd2265.out‚0 disk.sda1.out‚0 disk.sda.in‚ 0.05078125# memory_percent‚91.4571762084961 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 334.1484375 disk.sda2.in‚0 disk.drbd1716.out‚0 cpu‚25.36951864963146 network.sent‚ 5.335863e+06 disk.drbd2265.in‚0# disk.dm-0.out‚18745.56787109375& proc.memory.availableMB‚ 1399.65625 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 5497.45703125 disk.sda3.in‚ 0.05078125 disk.rbd1.out‚ 8.37158203125( disk.nvme0c0n1.out‚18745.56787109375 disk.dm-1.in‚ 0.05078125ãPúz÷‚(:’ ²ãÔê¹ß¯ gpu.0.gpu‚29.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚119.203' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.840600000000002 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™àznø‚(:ã ²ãÔÞŠß± disk.dm-1.out‚ 5526.15625 cpu‚25.367266979914763 disk.rbd0.in‚0 disk.sda3.in‚ 0.05078125! disk.nvme0c0n1.in‚ 334.1484375& disk.nvme0c0n1.out‚18746.244140625 disk.nvme0n1.in‚ 334.1484375$ disk.nvme0n1.out‚18746.244140625 disk.sda.in‚ 0.05078125% disk./.usageGB‚252.21463775634766 proc.cpu.threads‚17 disk.drbd2080.out‚0! disk.dm-0.out‚18746.244140625 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda3.out‚ 5526.15625 disk.rbd1.in‚0 disk.sda.out‚ 5526.15625 network.recv‚789779 disk.sda2.out‚0' proc.memory.availableMB‚ 1400.140625) disk./.usagePercent‚16.92633596200504 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0* proc.memory.percent‚16.441845893859863 network.sent‚ 5.341318e+06 disk.rbd1.out‚ 8.37158203125 disk.drbd2266.out‚0 disk.drbd2265.in‚0$ memory_percent‚91.45421981811523# proc.memory.rssMB‚ 2693.83203125 disk.rbd0.out‚0 disk.dm-0.in‚ 334.1484375 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.05078125PmëÔ2ù- _wandbruntime‚2117  _runtime‚2117×8‚8Èú _runtime‚2117.539047324 _step‚206 ' lr-AdamW/pg1‚0.00020704151713154504 ' lr-AdamW/pg2‚0.00020704151713154504 trainer/global_step‚2549 $ _timestamp‚1.788397492902178e+09¨>w¬ÍûÇ ' lr-AdamW/pg2‚0.00020704151713154504 trainer/global_step‚2549 $ _timestamp‚1.788397492902178e+09 _runtime‚2117.539047324 _step‚206 ' lr-AdamW/pg1‚0.00020704151713154504ÎÌô@ 2ü- _wandbruntime‚2117  _runtime‚2117nP?£Æ'ýÀO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚207 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51576880 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19638556 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14584067 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3659651592704 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85195770 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 $ _timestamp‚1.788397492902924e+09 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚75523253481984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3596696788480 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚79868931865088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚2117.539992528 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34223148 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3586160852081 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51556843 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚75523066479616 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚72006913656297 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33638927 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚75523253481984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34222621 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚72006731495537 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚2549 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3596527789056 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51576880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33638929 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3596696788480 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3659447299072 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚68420570643456 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚79868703631360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚76209280272384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚71926556693504 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚71926538690560 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19638554 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14584592 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚68420587682816 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3596527789056ù¥aGË'þÅO ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3586325973481 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚51556843 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚33638929 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚76209256332288 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚85215809 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚75523066479616 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚19638556 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚14584067 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3659651592704 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚85195770 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 $ _timestamp‚1.788397492902924e+09 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚75523253481984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3596696788480 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚79868931865088 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚2117.539992528 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34223148 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3586160852081 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚51556843 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚75523066479616 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚72006913656297 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚33638927 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚75523253481984 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚34222621 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚72006731495537 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚2549 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3596527789056 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚51576880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚33638929 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3596696788480 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3659447299072 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚68420570643456 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚79868703631360 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚76209280272384 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚71926556693504 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚71926538690560 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚19638554 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14584592 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚68420587682816 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3596527789056 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚207 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚51576880 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 I8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51576630 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚75551728982528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51596693 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚71953808055296 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85228216 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51596693 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33651588 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3598089939968 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51576630 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3660866718720 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚71953826058240 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3597920927232 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3597920927232 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34235341 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 K>DeviceStatsMonJÛ~iúitor.on_train_batch_end/allocation.all.allocated‚85248281 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34235848 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14589915 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚505 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79899266316800 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚68446513825792 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚75551915998208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317568 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚76238193220608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 trainer/global_step‚2549 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚71953826058240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33651588 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚507 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚68446530865152 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2118.243626654 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33651586 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206377472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚72034246003385 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3598089939968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85248281 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚79899035999232 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚71953808055296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85228216 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3661073096192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚75551915998208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33651586 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19645933 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14589410 _step‚208 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34235341 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚85248281 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚34235848 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚14589915 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚505 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚79899266316800 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚68446513825792 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚75551915998208 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317568 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚76238193220608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 trainer/global_step‚2549 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚71953826058240 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚33651588 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚507 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚68446530865152 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚2118.243626654 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚33651586 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206377472 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚72034246003385 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3598089939968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚85248281 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚79899035999232 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚71953808055296 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚85228216 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3661073096192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚75551915998208 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚33651586 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚19645933 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚14589410 _step‚208 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚51576630 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚75551728982528 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚51596693 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚71953808055296 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚85228216 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚51596693 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚33651588 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3598089939968 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚51576630 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3660866718720 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968ЧïÙ“2‚- _wandbruntime‚2118  _runtime‚2118:¤ sWƒÑ _runtime‚2118.245062144 .loss/FRt3/detection_loss‚2.482760701328516 .loss/FRt4/detection_loss‚2.629977681287905 /loss/FRt2/assignment_loss‚4.561445052807148 /loss/FBt2/detection_loss‚2.7963222791166866 trainer/global_step‚2549 _step‚209 0loss/FBt3/assignment_loss‚0.4883207529783249 .loss/FRt2/detection_loss‚3.443311544565054 /loss/FBt4/detection_loss‚0.8625858549623413 /loss/FRt3/assignment_loss‚3.528449258208275 0loss/FRt4/assignment_loss‚1.8597937604276145 1loss/FBt2/assignment_loss‚0.45915521565612993 1loss/FBt4/assignment_loss‚0.23840472443069266 epoch‚2 0loss/FRt1/assignment_loss‚3.3583592596379193 2loss/SRqqt3/assignment_loss‚1.3858752471705278 1loss/FBt1/assignment_loss‚0.43096622682017316 1loss/SRqqt1/detection_loss‚2.1224528043107553 $ _timestamp‚1.788397493608291e+09 2loss/SRqqt1/assignment_loss‚2.3698382915674965 1loss/SRqqt2/assignment_loss‚2.455476466152403 2loss/SRqqt4/assignment_loss‚1.3374724363932362 1loss/SRqqt2/detection_loss‚2.5673967644020363 0loss/SRqqt4/detection_loss‚1.082458641912256 .loss/FBt1/detection_loss‚2.672620540156084 .loss/FRt1/detection_loss‚2.517809296873483 0loss/SRqqt3/detection_loss‚1.598018121222655 /loss/FBt3/detection_loss‚3.2742079165246754 %loss/total_loss‚50.52347883891239>Ÿ»Q\„Ö 0loss/FRt4/assignment_loss‚1.8597937604276145 1loss/FBt2/assignment_loss‚0.45915521565612993 1loss/FBt4/assignment_loss‚0.23840472443069266 epoch‚2 0loss/FRt1/assignment_loss‚3.3583592596379193 2loss/SRqqt3/assignment_loss‚1.3858752471705278 1loss/FBt1/assignment_loss‚0.43096622682017316 1loss/SRqqt1/detection_loss‚2.1224528043107553 $ _timestamp‚1.788397493608291e+09 2loss/SRqqt1/assignment_loss‚2.3698382915674965 1loss/SRqqt2/assignment_loss‚2.455476466152403 2loss/SRqqt4/assignment_loss‚1.3374724363932362 1loss/SRqqt2/detection_loss‚2.5673967644020363 0loss/SRqqt4/detection_loss‚1.082458641912256 .loss/FBt1/detection_loss‚2.672620540156084 .loss/FRt1/detection_loss‚2.517809296873483 0loss/SRqqt3/detection_loss‚1.598018121222655 /loss/FBt3/detection_loss‚3.2742079165246754 %loss/total_loss‚50.52347883891239 _runtime‚2118.245062144 .loss/FRt3/detection_loss‚2.482760701328516 .loss/FRt4/detection_loss‚2.629977681287905 /loss/FRt2/assignment_loss‚4.561445052807148 /loss/FBt2/detection_loss‚2.7963222791166866 trainer/global_step‚2549 _step‚209 0loss/FBt3/assignment_loss‚0.4883207529783249 .loss/FRt2/detection_loss‚3.443311544565054 /loss/FBt4/detection_loss‚0.8625858549623413 /loss/FRt3/assignment_loss‚3.528449258208275ÑÚ}Ì…‚(:’ ÁãÔõȃ® gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚129.949' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.989800000000002 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fiß`v†‚(:ë ÁãÔ´Øð° disk.rbd1.out‚ 8.45751953125 disk.sda.in‚ 0.05078125 disk.sda2.in‚0$ memory_percent‚91.46044254302979 network.recv‚794002* disk./.usagePercent‚16.926401499933807 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 5557.5234375 cpu‚25.364963391833534 proc.cpu.threads‚17 disk.dm-0.in‚ 334.1484375! disk.dm-0.out‚18746.474609375 disk.sda2.out‚0 disk.dm-1.in‚ 0.05078125 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1399.12109375* proc.memory.percent‚16.441869735717773 disk.drbd1716.out‚0 disk.drbd2266.out‚0 network.sent‚ 5.347108e+06$ disk.nvme0n1.out‚18746.474609375" proc.memory.rssMB‚ 2693.8359375 disk.drbd2265.in‚0 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 334.1484375& disk.nvme0c0n1.out‚18746.474609375 disk.dm-1.out‚ 5557.5234375 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 334.1484375 disk.sda.out‚ 5557.5234375% disk./.usageGB‚252.21561431884766 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚ 0.05078125¶Üü’‡‚(:‡ ÐãԲċ® gpu.0.gpu‚21.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚103.382' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.6764 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©M*6|ˆ‚(:ñ ÐãÔŞ°' proc.memory.availableMB‚ 1399.609375 network.recv‚800084 disk.sda3.in‚ 0.05078125( disk.nvme0c0n1.out‚18746.84228515625 disk.sda2.out‚0* proc.memory.percent‚16.441869735717773) disk./.usagePercent‚16.92640175594134 disk.rbd1.out‚ 8.60595703125 disk.dm-1.in‚ 0.05078125" proc.memory.rssMB‚ 2693.8359375 cpu‚25.362910274065715 disk.nvme0n1.in‚ 334.1484375 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda1.out‚0% disk./.usageGB‚252.21561813354492 disk.drbd1716.out‚0 disk.drbd2080.in‚0# disk.dm-0.out‚18746.84228515625 disk.drbd2265.out‚0 disk.dm-0.in‚ 334.1484375 disk.sda1.in‚0 disk.rbd0.out‚0 network.sent‚ 5.455595e+06! disk.nvme0c0n1.in‚ 334.1484375 disk.sda.in‚ 0.05078125 disk.dm-1.out‚ 5590.73828125 disk.drbd2266.in‚0 disk.drbd2265.in‚0$ memory_percent‚91.45746231079102 disk.sda3.out‚ 5590.73828125 proc.cpu.threads‚17 disk.rbd1.in‚0& disk.nvme0n1.out‚18746.84228515625 disk.sda.out‚ 5590.73828125²E²Ô2‰- _wandbruntime‚2154  _runtime‚2154#ÒÚÉŠÃ _step‚210 ' lr-AdamW/pg1‚0.00020644566128042273 ' lr-AdamW/pg2‚0.00020644566128042273 trainer/global_step‚2599 % _timestamp‚1.7883975301533537e+09 _runtime‚2154.790345365½Ô~ˆÎ‹È ' lr-AdamW/pg1‚0.00020644566128042273 ' lr-AdamW/pg2‚0.00020644566128042273 trainer/global_step‚2599 % _timestamp‚1.7883975301533537e+09 _runtime‚2154.790345365 _step‚210Ò.Ü 2Œ- _wandbruntime‚2154  _runtime‚2154æÑŽÂÇ'ÁO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3666368111104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20007404 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚73290024930304 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚86821422 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76956393041408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34271879 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76956206039040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232428032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3666368111104 trainer/global_step‚2599 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚81385251978752 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚73373362430745 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52549545 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52569582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚76956206039040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚81385484406784 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3730556723200 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3666199111680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚52569582 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 % _timestamp‚1.7883975301541452e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚514 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3666199111680 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚34863371 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3730348235264 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34271879 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52549545 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14855965 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚52549545 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚14855965 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚76956393041408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34271879 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚76956206039040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232428032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3666368111104 trainer/global_step‚2599 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚81385251978752 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚73373362430745 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚52549545 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚52569582 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚76956206039040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160ÓŠ àS2- _wandbruntime‚2155  _runtime‚2155 ¸õ0Ñ&ËM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚73317276292096 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚510 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚81415582402560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474624 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚86853868 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚86853868 % _timestamp‚1.7883975308509214e+09 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚81415814817280 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34284536 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52589395 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3731974185472 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2599 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3657021835033 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚77683840631808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3657186956641 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86873933 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚512 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52569332 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20014781 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76984868541952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34875622 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34284538 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3667761262592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3667761262592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚73317294295040 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚69743672942592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34284536 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52569332 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _step‚212 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3667592249856 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3667761262592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚73317294295040 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚69743672942592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚34284536 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚52569332 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _step‚212 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3667592249856 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34284536 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚52589395 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3731974185472 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚2599 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3657021835033 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚77683840631808 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3657186956641 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 ODeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚86873933 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚512 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚52569332 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20014781 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚76984868541952 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚34875622 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚34284538 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3667761262592 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F9–2ž- _wandbruntime‚2191  _runtime‚2191VÀÎþÇ'ŸÁO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚74653493167104 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210585088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20376256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34904829 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚546 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35502456 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚79100551154688 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 trainer/global_step‚2649 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53562284 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34904827 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82901847647232 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚74739993365953 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚71014888877056 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚74740175526713 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88467113 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15126746 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚74653475164160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3736039433728 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79100575094784 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚71014905916416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3735870434304 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3725269610297 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53542247 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚74653493167104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34904829 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78389345598464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53542247 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3801296492544 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚544 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚82902082172416 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88447074 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35503002 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3725104488897 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚78389345598464 _step‚215 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234525184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _runtime‚2191.821669248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 IDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚78389345598464 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚53542247 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3801296492544 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚544 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚82902082172416 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚88447074 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 265683968 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚35503002 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3725104488897 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚78389345598464 _step‚215 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234525184 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _runtime‚2191.821669248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210585088 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20376256 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚34904829 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚546 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚35502456 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚79100551154688 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 trainer/global_step‚2649 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚53562284 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚34904827 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚82901847647232 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚74739993365953 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚71014888877056 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚74740175526713 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚88467113 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15126746 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚74653475164160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3736039433728 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚79100575094784 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚71014905916416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3735870434304 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3725269610297 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚53542247 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚74653493167104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚34904829×6ì¥i2¡- _wandbruntime‚2192  _runtime‚2192Réæ¢ËM NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚53582097 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚88479520 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3726658775049 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚78418195117056 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 trainer/global_step‚2649 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53562034 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3802721082880 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚74680744528896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚78418008101376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 % _timestamp‚1.7883975678840897e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79129464102912 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15132068 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚216 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88499585 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20383631 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228220416 _runtime‚2192.521562893 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20383633 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88499585 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3802925363200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53562034 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚74680744528896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35515188 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82932413406208 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚74767507873801 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3737263572480 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34917486 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34917488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚513 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚82932185185792 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >6DeviceStatsMon‡g+së itor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3726658775049 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚78418195117056 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 trainer/global_step‚2649 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚53562034 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 3802721082880 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚74680744528896 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚78418008101376 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 % _timestamp‚1.7883975678840897e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚79129464102912 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15132068 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 _step‚216 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚88499585 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20383631 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228220416 _runtime‚2192.521562893 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚20383633 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚88499585 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3802925363200 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚53562034 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚74680744528896 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚35515188 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚82932413406208 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚74767507873801 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3737263572480 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚34917486 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚34917488 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚513 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚82932185185792 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 O*"Î­È trainer/global_step‚2699 % _timestamp‚1.7883976045373337e+09 _runtime‚2229.174249442 _step‚218 ' lr-AdamW/pg1‚0.00020519221348407006 ' lr-AdamW/pg2‚0.00020519221348407006ÚdC®ñ2®- _wandbruntime‚2229  _runtime‚2229A(I–Ç'¯ÁO WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20745106 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚84418383227392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35537777 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3805541756928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚90092765 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15397001 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚84418619849728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚76106624301161 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 % _timestamp‚1.7883976045380473e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚80546222505984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _runtime‚2229.175081672 _step‚219 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚79822672160256 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90072726 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3805541756928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35537777 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚72312047993856 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚79822672160256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚510 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚76016943400960 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79822485157888 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚79822485157888 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54534949 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚2699 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚80546198565888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚72312065033216 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35537779 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3872397343744 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚76016961403904 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54534949 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36142615 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54554986 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚76016943400960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3794576307305 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3805710756352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36142105 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54554986 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76106806461921 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039–›3t °ÆO C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3805710756352 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15397509 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3872184661504 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚76016961403904 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚20745104 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682240 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15397001 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚84418619849728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚76106624301161 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 % _timestamp‚1.7883976045380473e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚80546222505984 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _runtime‚2229.175081672 _step‚219 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚79822672160256 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E8_ÿ¿h¶DeviceStatsMonitor.on_train_batch_start/active.all.freed‚90072726 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3805541756928 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚35537777 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚72312047993856 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚79822672160256 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚510 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚76016943400960 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚79822485157888 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚79822485157888 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚54534949 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚2699 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚80546198565888 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚72312065033216 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚35537779 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3872397343744 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚76016961403904 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚54534949 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36142615 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622336 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚54554986 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚76016943400960 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3794576307305 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3805710756352 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36142105 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚54554986 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚76106806461921 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚20745106 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚84418383227392 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚35537777 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3805541756928 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚508 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚90092765 IDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90125237 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35550436 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35550436 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚84448714836992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3806934895104 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76134138809009 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚76044212765696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15403018 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚80575111514112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90105172 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 trainer/global_step‚2699 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15402494 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚72338008215552 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36154975 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54574799 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚79851147660800 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚72337991176192 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3807103907840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚79851334676480 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35550438 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54554736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414720 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚76133956648041 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90125237 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76044230768640 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20752481 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54574799 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0Dé\GÕ&³ÏM C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚90105172 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 trainer/global_step‚2699 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15402494 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚72338008215552 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36154975 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚54574799 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚79851147660800 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚72337991176192 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3807103907840 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚79851334676480 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚35550438 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚54554736 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414720 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚76133956648041 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚90125237 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚76044230768640 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚20752481 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚54574799 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚90125237 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚35550436 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚35550436 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚84448714836992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3806934895104 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚76134138809009 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚76044212765696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15403018 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚80575111514112 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0Ü`ª7 2´- _wandbruntime‚2229  _runtime‚22296¸•›TµÎ /loss/FBt2/detection_loss‚2.9754193641922693 trainer/global_step‚2699 0loss/FBt4/assignment_loss‚0.1872972418418334 0loss/SRqqt2/detection_loss‚2.557887110216864 _step‚221 2loss/SRqqt1/assignment_loss‚1.8483414700047838 0loss/FBt2/assignment_loss‚0.5797051375080958 0loss/FBt3/assignment_loss‚0.4729061852988252 1loss/SRqqt4/detection_loss‚1.0259690322499309 -loss/FBt3/detection_loss‚3.01662378013134 /loss/FRt1/assignment_loss‚4.006027103825049 /loss/FRt3/assignment_loss‚3.580147303067721 /loss/FRt1/detection_loss‚2.3829109485853803 -loss/FRt2/detection_loss‚3.09950821527413 .loss/FRt3/detection_loss‚2.451997003494165 %loss/total_loss‚49.83851443099651 0loss/FRt4/assignment_loss‚1.7070411123611309 .loss/FBt1/detection_loss‚2.250114897886912 2loss/SRqqt2/assignment_loss‚2.1945960573081313 % _timestamp‚1.7883976052334352e+09 /loss/FBt4/detection_loss‚0.6683210785568979 epoch‚2 0loss/FRt2/assignment_loss‚5.3943396891866415 1loss/SRqqt3/assignment_loss‚1.702279865480306 1loss/SRqqt1/detection_loss‚2.2403119607104194 _runtime‚ 2229.87018129 1loss/FBt1/assignment_loss‚0.46023666113615036 1loss/SRqqt3/detection_loss‚1.8645163486924088 1loss/SRqqt4/assignment_loss‚1.390541482406358 /loss/FRt4/detection_loss‚1.7814753815807678`_¬CY¶Ó 1loss/SRqqt4/assignment_loss‚1.390541482406358 /loss/FRt4/detection_loss‚1.7814753815807678 /loss/FBt2/detection_loss‚2.9754193641922693 trainer/global_step‚2699 0loss/FBt4/assignment_loss‚0.1872972418418334 0loss/SRqqt2/detection_loss‚2.557887110216864 _step‚221 2loss/SRqqt1/assignment_loss‚1.8483414700047838 0loss/FBt2/assignment_loss‚0.5797051375080958 0loss/FBt3/assignment_loss‚0.4729061852988252 1loss/SRqqt4/detection_loss‚1.0259690322499309 -loss/FBt3/detection_loss‚3.01662378013134 /loss/FRt1/assignment_loss‚4.006027103825049 /loss/FRt3/assignment_loss‚3.580147303067721 /loss/FRt1/detection_loss‚2.3829109485853803 -loss/FRt2/detection_loss‚3.09950821527413 .loss/FRt3/detection_loss‚2.451997003494165 %loss/total_loss‚49.83851443099651 0loss/FRt4/assignment_loss‚1.7070411123611309 .loss/FBt1/detection_loss‚2.250114897886912 2loss/SRqqt2/assignment_loss‚2.1945960573081313 % _timestamp‚1.7883976052334352e+09 /loss/FBt4/detection_loss‚0.6683210785568979 epoch‚2 0loss/FRt2/assignment_loss‚5.3943396891866415 1loss/SRqqt3/assignment_loss‚1.702279865480306 1loss/SRqqt1/detection_loss‚2.2403119607104194 _runtime‚ 2229.87018129 1loss/FBt1/assignment_loss‚0.46023666113615036 1loss/SRqqt3/detection_loss‚1.8645163486924088Ýt›’·‚(:‡ ªãÔ÷ݘ® gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚113.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7068 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0f€ó!s¸‚(:è ªãÔŸŠÏ°! disk.nvme0c0n1.in‚ 334.1484375 disk.sda2.in‚0 disk.drbd2266.in‚0" disk.dm-0.out‚18748.6416015625 disk.drbd2080.out‚0 disk.rbd0.in‚0 network.recv‚827807' disk.nvme0c0n1.out‚18748.6416015625 disk.dm-1.out‚ 5777.546875 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 1399.46875 disk.rbd1.in‚0 disk.dm-1.in‚ 0.05078125 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda3.in‚ 0.05078125# proc.memory.rssMB‚ 2693.83984375 proc.cpu.threads‚17 disk.dm-0.in‚ 334.1484375 network.sent‚ 5.695413e+06* disk./.usagePercent‚16.926402011948873$ disk./.usageGB‚252.2156219482422% disk.nvme0n1.out‚18748.6416015625 disk.sda3.out‚ 5777.546875* proc.memory.percent‚16.441893577575684 disk.nvme0n1.in‚ 334.1484375 disk.sda.in‚ 0.05078125 disk.sda.out‚ 5777.546875 disk.drbd2266.out‚0$ memory_percent‚91.45832061767578 disk.sda2.out‚0 disk.rbd1.out‚ 8.76416015625 cpu‚25.350557248647135„Zšð¹‚(:… ¹ãÔÛŒ“® gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚115.13' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.026 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˜p»ø²º‚(:ò ¹ãÔª†± disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0* proc.memory.percent‚16.441893577575684 cpu‚25.348565518209185* disk./.usagePercent‚16.926405852061887% disk./.usageGB‚252.21567916870117 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0! disk.nvme0c0n1.in‚ 334.1484375' disk.nvme0c0n1.out‚18748.8251953125# proc.memory.rssMB‚ 2693.83984375 network.sent‚ 5.802864e+06 disk.sda3.in‚ 0.05078125 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.05078125 disk.sda1.in‚0) proc.memory.availableMB‚ 1399.44921875 network.recv‚833950 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 334.1484375" disk.dm-0.out‚18748.8251953125 disk.sda3.out‚ 5806.55078125 disk.rbd1.out‚ 8.85009765625 disk.sda.out‚ 5806.55078125 disk.dm-1.out‚ 5806.55078125 disk.sda1.out‚0 dTyî›Ëisk.nvme0n1.in‚ 334.1484375% disk.nvme0n1.out‚18748.8251953125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.drbd2080.out‚0 disk.sda.in‚ 0.05078125$ memory_percent‚91.45843982696533Ò´_D’»‚(:‡ ÈãÔ̪§® gpu.0.gpu‚32.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚125.106' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.0212 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Çdj}¼‚(:ò ÈãÔ°í’± disk.dm-1.out‚ 5830.21484375& proc.memory.availableMB‚ 1399.40625 cpu‚25.346607529059884 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 334.1484375( disk.nvme0c0n1.out‚18749.60888671875 disk.drbd1716.out‚0 disk.sda3.out‚ 5830.21484375 disk.rbd1.in‚0& disk.nvme0n1.out‚18749.60888671875 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.05078125 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.05078125# proc.memory.rssMB‚ 2693.83984375 network.sent‚ 5.808831e+06% disk./.usageGB‚252.21567916870117* proc.memory.percent‚16.441893577575684* disk./.usagePercent‚16.926405852061887 disk.drbd2265.in‚0# disk.dm-0.out‚18749.60888671875 disk.sda1.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 334.1484375 disk.sda.out‚ 5830.21484375 disk.sda2.in‚0 network.recv‚838201 disk.rbd1.out‚ 8.91259765625 disk.sda.in‚ 0.05078125$ memory_percent‚91.45870208740234 disk.drbd2266.out‚0 disk.nvme0n1.in‚ 334.1484375 disk.sda2.out‚0X2½- _wandbruntime‚2265  _runtime‚2265hI’—Ⱦ trainer/global_step‚2749 $ _timestamp‚1.788397641249705e+09 _runtime‚2265.886549325 _step‚222 ' lr-AdamW/pg1‚0.00020453488275548175 ' lr-AdamW/pg2‚0.00020453488275548175ÁÊKoÍ¿Ç ' lr-AdamW/pg1‚0.00020453488275548175 ' lr-AdamW/pg2‚0.00020453488275548175 trainer/global_step‚2749 $ _timestamp‚1.788397641249705e+09 _runtime‚2265.886549325 _step‚222Þaµ Œ2À- _wandbruntime‚2265  _runtime‚22651}kÆ'ÁÀO TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 trainer/global_step‚2749 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚81255811719680 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚73609207110656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36783709 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3943264711168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36170729 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3864048125713 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚81255811719680 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91718417 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214779392 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36170727 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚81255624717312 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55547688 _runtime‚2265.887523353 _step‚223 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55527651 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 $ _timestamp‚1.788397641250422e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55527651 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21113954 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚85934895908864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚77380429640704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15669753 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3864213247113 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36170729 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚77380411637760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3875382078976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55547688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36783174 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3943049931776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91698378 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3875382078976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21113956 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚85935134628352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0†çm®Ë'ÂÅO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚55547688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 IDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚36783174 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 3943049931776 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚91698378 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3875382078976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21113956 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚85935134628352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 trainer/global_step‚2749 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚81255811719680 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚73609207110656 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚36783709 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 3943264711168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36170729 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3864048125713 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚81255811719680 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚91718417 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214779392 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36170727 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚81255624717312 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚55547688 _runtime‚2265.887523353 _step‚223 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚55527651 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 $ _timestamp‚1.788397641250422e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚55527651 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21113954 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚85934895908864 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚77380429640704 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15669753 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3864213247113 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36170729 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚77380411637760 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3875382078976 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0ßeÞ5ð2Ã- _wandbruntime‚2266  _runtime‚2266ŒáƒÑ&ÄËM F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204280320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚77500587583249 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36795921 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36183386 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82020782865408 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3865602411865 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚81284287220224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚77407699005440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91730824 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36183388 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚497 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚77407681002496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚73635150292992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91750889 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚499 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36183388 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36183386 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 trainer/global_step‚2749 _step‚224 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55567501 % _timestamp‚1.7883976419397774e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚77407681002496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15675087 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3944678343168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77500769744217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3865437290257 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚77407699005440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3876606217728 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚82020758925312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91730824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21121331 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3876606217728 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚85965461208576 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚81284474235904 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚85965232988160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3876775230464 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55567501 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228220416 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 _runtime‚2266.577229784ÙLË!EÅÐM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.Eçv)‘&freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚81284474235904 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚85965232988160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3876775230464 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚55567501 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228220416 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 _runtime‚2266.577229784 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204280320 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚77500587583249 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚36795921 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 ODeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36183386 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚82020782865408 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3865602411865 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚81284287220224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚77407699005440 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚91730824 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36183388 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚497 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚77407681002496 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚73635150292992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚91750889 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚499 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36183388 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36183386 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 trainer/global_step‚2749 _step‚224 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚55567501 % _timestamp‚1.7883976419397774e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚77407681002496 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15675087 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 3944678343168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚77500769744217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3865437290257 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚77407699005440 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3876606217728 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚82020758925312 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚91730824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21121331 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3876606217728 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚85965461208576 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416àùGž/2Æ- _wandbruntime‚2266  _runtime‚2266U¼¼ÌWÇÑ 0loss/FRt1/assignment_loss‚3.5698428349061446 .loss/FRt4/detection_loss‚1.798588009694448 0loss/SRqqt2/detection_loss‚2.517018473148346 1loss/SRqqt4/detection_loss‚1.0185444567600885 /loss/FBt1/detection_loss‚2.6560022155443828 _runtime‚2266.578258776 /loss/FRt2/assignment_loss‚4.222735189256214 /loss/FRt3/assignment_loss‚3.587905891398166 2loss/SRqqt1/assignment_loss‚1.5945726723543234 2loss/SRqqt4/assignment_loss‚1.1915316582719484 0loss/FBt2/assignment_loss‚0.4665555531575722 1loss/FBt4/assignment_loss‚0.26864935692531244 0loss/SRqqt1/detection_loss‚2.130363918131306 &loss/total_loss‚48.309701724413486 trainer/global_step‚2749 /loss/FRt4/assignment_loss‚1.851598316660294 2loss/SRqqt2/assignment_loss‚2.1603269557158153 0loss/SRqqt3/detection_loss‚2.004179136384101 1loss/SRqqt3/assignment_loss‚2.059607459675698 0loss/FBt1/assignment_loss‚0.5010595398627629 epoch‚2 _step‚225 .loss/FRt3/detection_loss‚2.260561133318759 /loss/FBt2/detection_loss‚2.8592764692647115 % _timestamp‚1.7883976419415064e+09 1loss/FBt3/assignment_loss‚0.19868381023406984 /loss/FBt3/detection_loss‚3.1655786991119386 /loss/FRt2/detection_loss‚3.4763030722027732 -loss/FRt1/detection_loss‚2.05230344154618 /loss/FBt4/detection_loss‚0.6979134608881367…Væ\ÈÖ .loss/FRt4/detection_loss‚1.798588009694448 0loss/SRqqt2/detection_loss‚2.517018473148346 1loss/SRqqt4/detection_loss‚1.0185444567600885 /loss/FBt1/detection_loss‚2.6560022155443828 _runtime‚2266.578258776 /loss/FRt2/assignment_loss‚4.222735189256214 /loss/FRt3/assignment_loss‚3.587905891398166 2loss/SRqqt1/assignment_loss‚1.5945726723543234 2loss/SRqqt4/assignment_loss‚1.1915316582719484 0loss/FBt2/assignment_loss‚0.4665555531575722 1loss/FBt4/assignment_loss‚0.26864935692531244 0loss/SRqqt1/detection_loss‚2.130363918131306 &loss/total_loss‚48.309701724413486 trainer/global_step‚2749 /loss/FRt4/assignment_loss‚1.851598316660294 2loss/SRqqt2/assignment_loss‚2.1603269557158153 0loss/SRqqt3/detection_loss‚2.004179136384101 1loss/SRqqt3/assignment_loss‚2.059607459675698 0loss/FBt1/assignment_loss‚0.5010595398627629 epoch‚2 _step‚225 .loss/FRt3/detection_loss‚2.260561133318759 /loss/FBt2/detection_loss‚2.8592764692647115 % _timestamp‚1.7883976419415064e+09 1loss/FBt3/assignment_loss‚0.19868381023406984 /loss/FBt3/detection_loss‚3.1655786991119386 /loss/FRt2/detection_loss‚3.4763030722027732 -loss/FRt1/detection_loss‚2.05230344154618 /loss/FBt4/detection_loss‚0.6979134608881367 0loss/FRt1/assignment_loss‚3.5698428349061446á¨Ñ¶›É‚(: ×ãÔ†«™® gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚114.6' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.919999999999998 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤²xÊ‚(:í ×ãÔç¹°± network.sent‚ 5.815066e+06( disk./.usagePercent‚16.9264096921749% disk./.usageGB‚252.21573638916016 disk.rbd0.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2693.83984375 disk.dm-1.in‚ 0.05078125 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 8.91259765625 disk.nvme0n1.in‚ 334.1484375 disk.drbd2266.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 334.1484375& disk.nvme0n1.out‚18749.89013671875 network.recv‚843096 disk.drbd1716.in‚0 disk.sda3.out‚ 5854.49609375 disk.rbd1.in‚0( disk.nvme0c0n1.out‚18749.89013671875 disk.sda3.in‚ 0.05078125 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda.in‚ 0.05078125 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.sda2.out‚0$ memory_percent‚91.45889282226562 cpu‚25.34467190492356 disk.dm-1.out‚ 5854.49609375 disk.sda.out‚ 5854.49609375$ proc.memory.availableMB‚1399.375* proc.memory.percent‚16.441893577575684 disk.drbd2266.out‚0 disk.dm-0.in‚ 334.1484375# disk.dm-0.out‚18749.89013671875â!Ì$Ë‚(:’ æãÔ¡®§® gpu.0.gpu‚22.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚134.552' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.910400000000003 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸avzÌ‚(:ï æãÔ³óä° disk.sda1.out‚0 disk.nvme0n1.in‚ 334.1484375 disk.sda.in‚ 0.05078125 disk.rbd1.out‚ 8.94482421875* proc.memory.percent‚16.441893577575684% disk./.usageGB‚252.21567916870117 disk.dm-0.in‚ 334.1484375 disk.sda3.out‚ 5878.8203125' disk.nvme0c0n1.out‚18750.2001953125) proc.memory.availableMB‚ 1399.37890625 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.05078125 disk.sda1.in‚0% disk.nvme0n1.out‚18750.2001953125# proc.memory.rssMB‚ 2693.83984375 disk.sda2.out‚0 disk.drbd2265.out‚0" disk.dm-0.out‚18750.2001953125 disk.rbd1.in‚0$ memory_percent‚91.45886898040771 cpu‚25.342672507998824 disk.drbd1716.out‚0 disk.sda3.in‚ 0.05078125! disk.nvme0c0n1.in‚ 334.1484375 network.sent‚ 5.922808e+06 network.recv‚848642 disk.sda2.in‚0 disk.dm-1.out‚ 5878.8203125 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0* disk./.usagePercent‚16.926405852061887 disk.sda.out‚ 5878.8203125 disk.drbd2265.in‚0dÞí;2Í- _wandbruntime‚2302  _runtime‚2302©ÌL8ÉÎà ' lr-AdamW/pg2‚0.00020385732870791787 trainer/global_step‚2799 % _timestamp‚1.7883976782875211e+09 _runtime‚2302.924432194 _step‚226 ' lr-AdamW/pg1‚0.00020385732870791787dŽuÎÏÈ % _timestamp‚1.7883976782875211e+09 _runtime‚2302.924432194 _step‚226 ' lr-AdamW/pg1‚0.00020385732870791787 ' lr-AdamW/pg2‚0.00020385732870791787 trainer/global_step‚2799â›öÝ2Ð- _wandbruntime‚2302  _runtime‚23025ö²ÎÇ'ÑÁO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37425302 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21482804 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93344069 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93324030 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3945053401600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚78743879874560 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚83437517328384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚78839886171577 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36803679 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82688764276736 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37425799 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3933519944121 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4014193042944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚82688764276736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3933685065521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78743879874560 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚87451710371328 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚497 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15942498 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚83437493388288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚227 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36803677 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3945053401600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682240 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚78743897877504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 % _timestamp‚1.7883976782882814e+09 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82688951279104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36803679 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _runtime‚2302.925418866 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56540390 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21482806 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚495 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3944884402176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15942993 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944s÷¤‚<ÒÆO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 3944884402176 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4014193042944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚82688764276736 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 3933685065521 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚78743879874560 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚87451710371328 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚497 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚15942498 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚83437493388288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚227 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚36803677 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 3945053401600 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682240 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚78743897877504 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622336 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 % _timestamp‚1.7883976782882814e+09 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚82688951279104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚36803679 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _runtime‚2302.925418866 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚56540390 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21482806 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚495 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 3944884402176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚15942993 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚37425302 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21482804 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚93344069 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚93324030 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 3945053401600 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚78743879874560 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚83437517328384 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitoœ¤*r.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚78839886171577 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚36803679 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚82688764276736 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚37425799 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 3933519944121 IDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93356476 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93376541 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚78867400679425 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3935074230273 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚78771149239296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56540140 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3934909108665 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚82717613795328 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15947691 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 $ _timestamp‚1.788397678990262e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15948165 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216863232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚78771149239296 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3946277540352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240803328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚74932326449152 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56540140 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4015609065472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56560203 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚87482039342080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚78771167242240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36816336 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _runtime‚2303.627628715 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚78867218518457 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36816338 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3946277540352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93356476 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _step‚228 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93376541 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37438348 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56560203 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21490183 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚82717426779648 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37437872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚74932309409792 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚476 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚82717613795328 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚87481798538752 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83466430276608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 trainer/global_step‚2799‘fx«Õ&ÕÏM F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚74932309409792 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚476 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚82717613795328 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚87481798538752 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚83466430276608 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 trainer/global_step‚2799 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚78771167242240 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚36816336 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 3946446553088 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 3946446553088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21490181 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚83466406336512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4015392202240 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚474 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚36816338 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚93356476 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚93376541 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚78867400679425 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 3935074230273 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚78771149239296 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚56540140 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 3934909108665 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚82717613795328 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚15947691 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 $ _timestamp‚1.788397678990262e+09 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚15948165 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216863232 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚78771149239296 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 3946277540352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240803328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚74932326449152 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚56540140 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4015609065472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚56560203 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚87482039342080 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚78771167242240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚36816336 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _runtime‚2303.627628715 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚78867218518457 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚36816338 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 3946277540352 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚93356476 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _step‚228 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚93376541 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚37438348 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚56560203 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21490183 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚82717426779648 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚37437872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ä£LZ2Ö- _wandbruntime‚2303  _runtime‚23033^¥Y×Ó /loss/FRt4/detection_loss‚1.9142209342065848 trainer/global_step‚2799 _step‚229 1loss/SRqqt3/assignment_loss‚2.030261170405608 0loss/FBt1/assignment_loss‚0.6988729459957944 1loss/SRqqt4/detection_loss‚1.4270214445766856 /loss/FBt1/detection_loss‚2.6086453133159213 1loss/FBt4/assignment_loss‚0.19848574176463069 0loss/FRt4/assignment_loss‚1.5991088658008936 2loss/SRqqt1/assignment_loss‚1.7342074435603405 1loss/SRqqt4/assignment_loss‚1.365706048814244 0loss/SRqqt1/detection_loss‚1.846788480345692 .loss/FRt3/detection_loss‚2.792400559081751 % _timestamp‚1.7883976789921083e+09 _runtime‚2303.628922908 .loss/FRt2/detection_loss‚3.656268894672394 /loss/FRt2/assignment_loss‚4.955129518228419 2loss/SRqqt2/assignment_loss‚2.4986319777212644 epoch‚2 0loss/FRt1/assignment_loss‚3.3303836085358443 1loss/SRqqt2/detection_loss‚2.5976210675741496 /loss/FRt3/assignment_loss‚3.736392617225647 0loss/FBt3/assignment_loss‚0.3953392248094663 1loss/SRqqt3/detection_loss‚2.0343525868195753 /loss/FBt3/detection_loss‚2.8752873639265695 %loss/total_loss‚50.67908431704212 /loss/FRt1/detection_loss‚2.2211708377818673 /loss/FBt2/detection_loss‚2.7145558645327887 /loss/FBt4/detection_loss‚0.8486836692501402 0loss/FBt2/assignment_loss‚0.5995481380958457MRDl^ØØ .loss/FRt3/detection_loss‚2.792400559081751 % _timestamp‚1.7883976789921083e+09 _runtime‚2303.628922908 .loss/FRt2/detection_loss‚3.656268894672394 /loss/FRt2/assignment_loss‚4.955129518228419 2loss/SRqqt2/assignment_loss‚2.4986319777212644 epoch‚2 0loss/FRt1/assignment_loss‚3.3303836085358443 1loss/SRqqt2/detection_loss‚2.5976210675741496 /loss/FRt3/assignment_loss‚3.736392617225647 0loss/FBt3/assignment_loss‚0.3953392248094663 1loss/SRqqt3/detection_loss‚2.0343525868195753 /loss/FBt3/detection_loss‚2.8752873639265695 %loss/total_loss‚50.67908431704212 /loss/FRt1/detection_loss‚2.2211708377818673 /loss/FBt2/detection_loss‚2.7145558645327887 /loss/FBt4/detection_loss‚0.8486836692501402 0loss/FBt2/assignment_loss‚0.5995481380958457 /loss/FRt4/detection_loss‚1.9142209342065848 trainer/global_step‚2799 _step‚229 1loss/SRqqt3/assignment_loss‚2.030261170405608 0loss/FBt1/assignment_loss‚0.6988729459957944 1loss/SRqqt4/detection_loss‚1.4270214445766856 /loss/FBt1/detection_loss‚2.6086453133159213 1loss/FBt4/assignment_loss‚0.19848574176463069 0loss/FRt4/assignment_loss‚1.5991088658008936 2loss/SRqqt1/assignment_loss‚1.7342074435603405 1loss/SRqqt4/assignment_loss‚1.365706048814244 0loss/SRqqt1/detection_loss‚1.846788480345692å'Åhƒ’Ù‚(:‡ õãÔ—逮 gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚126.757' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.3514 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿7û¸oÚ‚(:ä õãÔôð° disk.rbd0.in‚0 disk.sda1.out‚0' proc.memory.availableMB‚ 1399.390625% disk./.usageGB‚252.21570205688477 disk.dm-1.out‚ 5907.6484375 disk.drbd2265.out‚0 disk.rbd1.out‚ 8.94482421875* proc.memory.percent‚16.441893577575684* disk./.usagePercent‚16.926407388107094 disk.sda.out‚ 5907.6484375 disk.drbd2265.in‚0 disk.dm-0.in‚ 334.1484375 disk.sda3.in‚ 0.05078125 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 334.1484375$ disk.nvme0c0n1.out‚ 18750.5078125 disk.nvme0n1.in‚ 334.1484375$ memory_percent‚91.45879745483398 proc.cpu.threads‚17 network.sent‚ 5.928669e+06 disk.sda.in‚ 0.05078125 disk.sda2.in‚0 disk.drbd2266.out‚0 network.recv‚852786 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.dm-0.out‚ 18750.5078125 disk.sda3.out‚ 5907.6484375 cpu‚25.340790100638056 disk.drbd1716.in‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2693.83984375" disk.nvme0n1.out‚ 18750.5078125 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 0.05078125+‘Q’Û‚(:‡ „‘ãÔñ±„® gpu.0.gpu‚33.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚132.667' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.5334 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê_ŸqÜ‚(:æ „‘ãÔüˆ»° disk.dm-0.out‚18750.88671875 disk.sda3.in‚ 0.05078125 disk.sda3.out‚ 5935.07421875% disk.nvme0c0n1.out‚18750.88671875 disk.nvme0n1.in‚ 334.1484375 disk.sda2.in‚0 disk.drbd1716.in‚0 network.sent‚ 6.037083e+06% disk./.usageGB‚252.21568298339844 disk.drbd1716.out‚0 disk.dm-1.out‚ 5935.07421875* proc.memory.percent‚16.441893577575684 network.recv‚858845* disk./.usagePercent‚16.926406108069422 disk.drbd2080.out‚0 disk.sda.out‚ 5935.07421875 disk.sda2.out‚0 disk.sda1.out‚0# disk.nvme0n1.out‚18750.88671875 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0' proc.memory.availableMB‚ 1399.109375 proc.cpu.threads‚17 disk.dm-0.in‚ 334.1484375 disk.rbd1.out‚ 9.0078125# proc.memory.rssMB‚ 2693.83984375 disk.rbd0.out‚0 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 334.1484375$ memory_percent‚91.46051406860352 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.05078125 disk.dm-1.in‚ 0.05078125 cpu‚25.338837049679835 disk.drbd2265.in‚0É‘ÁV2Ý- _wandbruntime‚2339  _runtime‚2339XqpÇÞÁ & lr-AdamW/pg1‚0.0002031596925426264 & lr-AdamW/pg2‚0.0002031596925426264 trainer/global_step‚2849 % _timestamp‚1.7883977152374926e+09 _runtime‚2339.874347008 _step‚230¥WwûÌßÆ & lr-AdamW/pg1‚0.0002031596925426264 & lr-AdamW/pg2‚0.0002031596925426264 trainer/global_step‚2849 % _timestamp‚1.7883977152374926e+09 _runtime‚2339.874347008 _step‚230æDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84121903836160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4014724724224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16212667 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚80206699267545 _runtime‚2339.875226508 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚88968246010880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4084872783360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚84122090838528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38064323 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚80107366114304 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4002991762529 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94949682 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_byte~õ©‚s.all.freed‚84121903836160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚76203525344256 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4014555724800 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚2849 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232428032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚84883164739584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21851656 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚88968013582848 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚80107348111360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 % _timestamp‚1.7883977152381802e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16212125 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚80107348111360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4003156883929 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚76203542383616 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57533092 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37436629 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57513055 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21851654 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _step‚231 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚80107366114304 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37436627 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38063779 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37436627 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚84122090838528 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94969721 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37436629 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80206517106785 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57533092 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4014724724224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199JôÔÌ'âÆO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4014555724800 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚2849 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232428032 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚84883164739584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚21851656 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚88968013582848 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚80107348111360 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 % _timestamp‚1.7883977152381802e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16212125 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚80107348111360 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4003156883929 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚76203542383616 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚57533092 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚37436629 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚57513055 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚21851654 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _step‚231 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚80107366114304 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚37436627 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚38063779 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚37436627 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚84122090838528 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚94969721 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚37436629 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚80206517106785 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚57533092 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4014724724224 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚84121903836160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4014724724224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16212667 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚80206699267545 _runtime‚2339.875226508 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚88968246010880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4084872783360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚84122090838528 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚38064323 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚80107366114304 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4002991762529 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚94949682 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚84121903836160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚76203525344256çžQö’ã‚(:‡ “‘ãÔÉÀµ® gpu.0.gpu‚28.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚134.263' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.8526 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ï@jä‚(:ß “‘ãÔÏ˧± disk.rbd1.in‚0 cpu‚25.336986075962827 disk.drbd2080.in‚0 disk.dm-0.out‚18751.11328125 disk.sda3.out‚ 5962.140625* disk./.usagePercent‚16.926409948182435% disk./.usageGB‚252.21574020385742 disk.dm-1.out‚ 5962.140625% disk.nvme0c0n1.out‚18751.11328125' proc.memory.availableMB‚ 1399.109375 disk.sda.out‚ 5962.140625 disk.sda3.in‚ 0.05078125 disk.rbd1.out‚ 9.0078125! disk.nvme0c0n1.in‚ 334.1484375# proc.memory.rssMB‚ 2693.83984375 disk.drbd2266.out‚0 network.recv‚863316* proc.memory.percent‚16.441893577575684 disk.drbd2265.out‚0 disk.dm-0.in‚ 334.1484375 disk.sda1.in‚0 disk.nvme0n1.in‚ 334.1484375 disk.sda.in‚ 0.05078125 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0$ memory_percent‚91.46051406860352 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.drbd2080.out‚0 network.sent‚ 6.04331e+06# disk.nvme0n1.out‚18751.11328125 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.05078125ÞÛQ"2å- _wandbruntime‚2340  _runtime‚2340‡lõÐ&æÊM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57552905 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4015948862976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4004546048681 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84150753354752 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95002193 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 $ _timestamp‚1.788397715935297e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚80134617476096 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚80134635479040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21859033 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚84912077687808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80233849453665 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76229485565952 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94982128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94982128 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80234031614633 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚84912053747712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚232 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37449286 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21859031 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76229468526592 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4086501260800 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4016117875712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16217966 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57552905 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95002193 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚84150566339072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚2849 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38076472 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84150753354752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚80134617476096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16217441 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚88998578948608 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38076999 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4016117875712‚8ƒ çÏM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚80233849453665 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚76229485565952 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚94982128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚94982128 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚80234031614633 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚84912053747712 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚232 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚37449286 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚21859031 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚76229468526592 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4086501260800 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4016117875712 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16217966 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚57552905 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚95002193 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚84150566339072 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚2849 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38076472 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571776 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚84150753354752 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚80134617476096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16217441 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚88998578948608 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38076999 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4016117875712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚57552905 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4015948862976 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4004546048681 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚84150753354752 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚95002193 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 $ _timestamp‚1.788397715935297e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚80134617476096 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚80134635479040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚21859033 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚84912077687808 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096è\/\2è- _wandbruntime‚2340  _runtime‚23408¾@,SéÍ 1loss/SRqqt4/detection_loss‚1.2800967732874247 1loss/SRqqt4/assignment_loss‚1.456972621811124 /loss/FRt2/detection_loss‚2.9057090200226883 1loss/SRqqt1/detection_loss‚1.8450038527017054 0loss/FRt3/assignment_loss‚3.6075562953948976 trainer/global_step‚2849 _runtime‚2340.573994255 _step‚233 epoch‚2 /loss/FRt2/assignment_loss‚4.134872259764836 2loss/SRqqt1/assignment_loss‚1.5747711501367714 0loss/SRqqt3/assignment_loss‚2.10696132555604 /loss/FBt1/detection_loss‚3.2999595871993472 /loss/FBt2/detection_loss‚2.7008160021569996 /loss/FRt1/assignment_loss‚4.309835785744237 0loss/FBt2/assignment_loss‚0.2900800490749336 0loss/FBt3/assignment_loss‚0.5967369304437723 .loss/FRt1/detection_loss‚2.054480079342337 .loss/FRt3/detection_loss‚2.671124806255102 /loss/FRt4/detection_loss‚2.0240964223191424 % _timestamp‚1.7883977159371862e+09 0loss/FRt4/assignment_loss‚2.4857928876606925 /loss/FBt4/detection_loss‚1.0443459188385324 %loss/total_loss‚50.98656344520586 1loss/SRqqt2/assignment_loss‚2.172569417184399 1loss/FBt4/assignment_loss‚0.12878919592445082 /loss/SRqqt3/detection_loss‚2.23132760822773 -loss/FBt3/detection_loss‚3.07702568599156 0loss/FBt1/assignment_loss‚0.3556373934261501 0loss/SRqqt2/detection_loss‚2.632002376740979¶’KXêÒ 0loss/FRt3/assignment_loss‚3.6075562953948976 trainer/global_step‚2849 _runtime‚2340.573994255 _step‚233 epoch‚2 /loss/FRt2/assignment_loss‚4.134872259764836 2loss/SRqqt1/assignment_loss‚1.5747711501367714 0loss/SRqqt3/assignment_loss‚2.10696132555604 /loss/FBt1/detection_loss‚3.2999595871993472 /loss/FBt2/detection_loss‚2.7008160021569996 /loss/FRt1/assignment_loss‚4.309835785744237 0loss/FBt2/assignment_loss‚0.2900800490749336 0loss/FBt3/assignment_loss‚0.5967369304437723 .loss/FRt1/detection_loss‚2.054480079342337 .loss/FRt3/detection_loss‚2.671124806255102 /loss/FRt4/detection_loss‚2.0240964223191424 % _timestamp‚1.7883977159371862e+09 0loss/FRt4/assignment_loss‚2.4857928876606925 /loss/FBt4/detection_loss‚1.0443459188385324 %loss/total_loss‚50.98656344520586 1loss/SRqqt2/assignment_loss‚2.172569417184399 1loss/FBt4/assignment_loss‚0.12878919592445082 /loss/SRqqt3/detection_loss‚2.23132760822773 -loss/FBt3/detection_loss‚3.07702568599156 0loss/FBt1/assignment_loss‚0.3556373934261501 0loss/SRqqt2/detection_loss‚2.632002376740979 1loss/SRqqt4/detection_loss‚1.2800967732874247 1loss/SRqqt4/assignment_loss‚1.456972621811124 /loss/FRt2/detection_loss‚2.9057090200226883 1loss/SRqqt1/detection_loss‚1.8450038527017054éÆgà‘ë‚(:† ¢‘ãÔý³ÿ­ gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚130.245' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.049 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×ê€ì‚(:õ ¢‘ãÔÜ̳°* disk./.usagePercent‚16.926406108069422 disk.sda.in‚ 0.05078125 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 network.sent‚ 6.139773e+06 disk.nvme0n1.in‚ 334.1484375 disk.sda2.out‚0 disk.dm-1.out‚ 5988.67578125# proc.memory.rssMB‚ 2693.83984375& disk.nvme0n1.out‚18751.43896484375 disk.drbd2080.in‚0 disk.dm-0.in‚ 334.1484375 disk.sda3.out‚ 5988.67578125 disk.dm-1.in‚ 0.05078125 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0% disk./.usageGB‚252.21568298339844 disk.drbd1716.out‚0 disk.sda3.in‚ 0.05078125 disk.rbd1.in‚0 cpu‚25.335197480904533# disk.dm-0.out‚18751.43896484375* proc.memory.percent‚16.441893577575684! disk.nvme0c0n1.in‚ 334.1484375 disk.sda.out‚ 5988.67578125 disk.rbd0.in‚0 disk.sda1.out‚0$ memory_percent‚91.46044254302979 proc.cpu.threads‚17 network.recv‚868921( disk.nvme0c0n1.out‚18751.43896484375 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 9.07080078125) proc.memory.availableMB‚ 1399.12109375M24í‚(:’ ±‘ãÔÏ® gpu.0.gpu‚30.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚133.733' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.746599999999997 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆÁRÑzî‚(:ï ±‘ãÔ”þ×± disk.drbd2080.out‚0 disk.rbd1.out‚ 9.15673828125 disk.sda.out‚ 6013.2109375 disk.drbd1716.out‚0$ disk./.usageGB‚252.2150115966797 disk.sda3.out‚ 6013.2109375 disk.drbd1716.in‚0 disk.dm-1.out‚ 6013.2109375# memory_percent‚91.4604663848877 disk.rbd0.out‚0 disk.sda1.out‚0& disk.nvme0n1.out‚18751.65771484375 disk.sda2.in‚0# proc.memory.rssMB‚ 2693.83984375 disk.rbd1.in‚0 proc.cpu.threads‚17# disk.dm-0.out‚18751.65771484375( disk.nvme0c0n1.out‚18751.65771484375 disk.nvme0n1.in‚ 334.1484375 disk.sda.in‚ 0.05078125 disk.drbd2266.in‚0 disk.drbd2265.in‚0* proc.memory.percent‚16.441893577575684 network.sent‚ 6.237735e+06 disk.drbd2080.in‚0 disk.sda3.in‚ 0.05078125 disk.dm-1.in‚ 0.05078125 disk.drbd2265.out‚0 disk.sda2.out‚0 network.recv‚874855* disk./.usagePercent‚16.926361050743395 disk.dm-0.in‚ 334.1484375 disk.sda1.in‚0 cpu‚25.333383083359827 disk.rbd0.in‚0! disk.nvme0c0n1.in‚ 334.1484375 disk.drbd2266.out‚0( proc.memory.availableMB‚ 1399.1171875g{2ï- _wandbruntime‚2376  _runtime‚2376YÈ6Èð _step‚234 ' lr-AdamW/pg1‚0.00020244211964593847 ' lr-AdamW/pg2‚0.00020244211964593847 trainer/global_step‚2899 $ _timestamp‚1.788397752209017e+09 _runtime‚2376.845863692Åf ÝÍñÇ _runtime‚2376.845863692 _step‚234 ' lr-AdamW/pg1‚0.00020244211964593847 ' lr-AdamW/pg2‚0.00020244211964593847 trainer/global_step‚2899 $ _timestamp‚1.788397752209017e+09ê˜/÷2ò- _wandbruntime‚2376  _runtime‚2376àÆ'óÀO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚81470834351104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4084227047424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16484458 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚494 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚81573330202753 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96575334 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16483964 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22220504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚77500684461056 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38069577 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚81470816348160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4084396046848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚77500701500416 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38069579 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85555043395584 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚81470816348160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4084396046848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22220506 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚86328812150784 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚81470834351104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38069579 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚90484736605184 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4072463580937 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58505757 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚86328788210688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4084227047424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58505757 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ÏÒOˆ$ôÅO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚81573330202753 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚96575334 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16483964 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22220504 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚77500684461056 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚38069577 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚81470816348160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4084396046848 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚77500701500416 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚38069579 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚85555043395584 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚81470816348160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4084396046848 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22220506 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚86328812150784 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚81470834351104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38069579 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚90484736605184 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4072463580937 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚58505757 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚86328788210688 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 IDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4084227047424 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚58505757 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚~%DC‚ 5041553408 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚81470834351104 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4084227047424 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16484458 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚494 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217ë³åý•2õ- _wandbruntime‚2377  _runtime‚2377$ÝÈÑ&öËM TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚38082236 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚96607780 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571776 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38082236 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22227881 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16489857 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58545607 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 _step‚236 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚81600480388873 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 trainer/global_step‚2899 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚90515070243328 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81498085712896 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4073852745481 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚81498085712896 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 % _timestamp‚1.7883977528973043e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58525544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511872 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96627845 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚85583705898496 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚90514835731456 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚86357701158912 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4157345144320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38717232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58525544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4157134572544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚2377.534853665 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58545607 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38082238 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96627845 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38082238 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22227883 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚77526644682752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4085789198336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38717740 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚85583892914176 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚81498103715840 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚81600662549841 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚86357701158912 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4157345144320 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚38717232 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚58525544 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4157134572544 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚2377.534853665 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚58545607 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38082238 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚96627845 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚38082238 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22227883 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚77526644682752 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4085789198336 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚38717740 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚85583892914176 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚81498103715840 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚81600662549841 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22227881 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚16489857 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚58545607 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 _step‚236 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚81600480388873 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 trainer/global_step‚2899 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚90515070243328 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚81498085712896 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4073852745481 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚81498085712896 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 % _timestamp‚1.7883977528973043e+09 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚58525544 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511872 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚96627845 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚85583705898496 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚90514835731456 EØVùÐ 0loss/FRt4/assignment_loss‚1.9824363986651103 2loss/SRqqt2/assignment_loss‚2.3282371257480823 0loss/FBt3/assignment_loss‚0.5466270742908819 % _timestamp‚1.7883977528993537e+09 _runtime‚2377.536091361 0loss/FRt1/assignment_loss‚4.1105989924183595 2loss/SRqqt3/assignment_loss‚1.7617041237176734 0loss/FBt1/assignment_loss‚0.7656793495019277 .loss/FRt1/detection_loss‚2.200197771743492 /loss/FRt2/detection_loss‚2.9857653379440308 .loss/SRqqt1/detection_loss‚1.9765731809186 /loss/FBt3/detection_loss‚2.9659341126680374 0loss/FRt3/assignment_loss‚3.2492972132232456 0loss/FBt2/assignment_loss‚0.2829749643278774 %loss/total_loss‚49.44418577824761 2loss/SRqqt1/assignment_loss‚1.4942276425221388 0loss/SRqqt4/assignment_loss‚1.25045361186588 1loss/SRqqt2/detection_loss‚2.3013818185580406 .loss/FBt2/detection_loss‚3.137297516067823 1loss/SRqqt4/detection_loss‚1.6125802942101033 trainer/global_step‚2899 /loss/FRt2/assignment_loss‚4.787800135283635 0loss/FBt4/assignment_loss‚0.2025578370266107 1loss/SRqqt3/detection_loss‚2.0616838551582175 epoch‚2 .loss/FRt3/detection_loss‚2.391148188047939 /loss/FRt4/detection_loss‚1.8567791313484863 /loss/FBt1/detection_loss‚2.4361152119106717 /loss/FBt4/detection_loss‚0.7561348910807454 _step‚237C¶[úÕ trainer/global_step‚2899 /loss/FRt2/assignment_loss‚4.787800135283635 0loss/FBt4/assignment_loss‚0.2025578370266107 1loss/SRqqt3/detection_loss‚2.0616838551582175 epoch‚2 .loss/FRt3/detection_loss‚2.391148188047939 /loss/FRt4/detection_loss‚1.8567791313484863 /loss/FBt1/detection_loss‚2.4361152119106717 /loss/FBt4/detection_loss‚0.7561348910807454 _step‚237 0loss/FRt4/assignment_loss‚1.9824363986651103 2loss/SRqqt2/assignment_loss‚2.3282371257480823 0loss/FBt3/assignment_loss‚0.5466270742908819 % _timestamp‚1.7883977528993537e+09 _runtime‚2377.536091361 0loss/FRt1/assignment_loss‚4.1105989924183595 2loss/SRqqt3/assignment_loss‚1.7617041237176734 0loss/FBt1/assignment_loss‚0.7656793495019277 .loss/FRt1/detection_loss‚2.200197771743492 /loss/FRt2/detection_loss‚2.9857653379440308 .loss/SRqqt1/detection_loss‚1.9765731809186 /loss/FBt3/detection_loss‚2.9659341126680374 0loss/FRt3/assignment_loss‚3.2492972132232456 0loss/FBt2/assignment_loss‚0.2829749643278774 %loss/total_loss‚49.44418577824761 2loss/SRqqt1/assignment_loss‚1.4942276425221388 0loss/SRqqt4/assignment_loss‚1.25045361186588 1loss/SRqqt2/detection_loss‚2.3013818185580406 .loss/FBt2/detection_loss‚3.137297516067823 1loss/SRqqt4/detection_loss‚1.6125802942101033íî˜YÖû‚(:’ À‘ãÔ¶¡‡® gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚107.576' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.515199999999997 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0löruü‚(:ê À‘ãÔž’¿° disk.drbd2265.out‚0 disk.sda1.out‚0* proc.memory.percent‚16.441893577575684 network.recv‚878820$ disk./.usageGB‚252.2159881591797 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.sda3.out‚ 6038.2734375! disk.nvme0c0n1.in‚ 334.1484375# proc.memory.rssMB‚ 2693.83984375 disk.sda3.in‚ 0.05078125( proc.memory.availableMB‚ 1399.2265625 cpu‚25.33164524250952 proc.cpu.threads‚17 network.sent‚ 6.24357e+06) disk./.usagePercent‚16.92642658867216 disk.rbd0.out‚0 disk.dm-0.in‚ 334.1484375$ memory_percent‚91.45979881286621 disk.sda.out‚ 6038.2734375 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 9.21923828125 disk.nvme0n1.in‚ 334.1484375 disk.sda.in‚ 0.05078125 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.dm-1.in‚ 0.05078125 disk.drbd2266.out‚0 disk.rbd0.in‚0% disk.nvme0n1.out‚18751.9345703125 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 6038.2734375" disk.dm-0.out‚18751.9345703125 disk.sda1.in‚0' disk.nvme0c0n1.out‚18751.9345703125(Mg’ý‚(:‡ Ï‘ãÔñ•œ® gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚113.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7068 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”5Eþ‚(:ô Ï‘ãÔ‰¸ÿ° disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd1.in‚0 disk.dm-1.in‚ 0.05078125 disk.drbd2265.in‚0# proc.memory.rssMB‚ 2693.83984375* disk./.usagePercent‚16.926366938916683 disk.rbd1.out‚ 9.30517578125 disk.nvme0n1.in‚ 334.1484375 disk.sda.in‚ 0.05078125 disk.drbd1716.out‚0 disk.drbd2266.out‚0$ memory_percent‚91.46502017974854 disk.sda1.out‚0 disk.sda3.in‚ 0.05078125 network.sent‚ 6.352272e+06( disk.nvme0c0n1.out‚18752.24755859375 disk.sda2.out‚0 disk.dm-1.out‚ 6064.34765625 disk.drbd2266.in‚0 cpu‚25.329892120340322$ disk./.usageGB‚252.2150993347168 disk.rbd0.out‚0# disk.dm-0.out‚18752.24755859375 disk.rbd0.in‚0 disk.sda3.out‚ 6064.34765625 disk.drbd1716.in‚0* proc.memory.percent‚16.441893577575684! disk.nvme0c0n1.in‚ 334.1484375& disk.nvme0n1.out‚18752.24755859375 disk.sda2.in‚0) proc.memory.availableMB‚ 1398.37109375 network.recv‚885209 disk.dm-0.in‚ 334.1484375 disk.sda.out‚ 6064.34765625uªx2ÿ- _wandbruntime‚2413  _runtime‚2413/Ó£îÇ€Á _runtime‚2413.729555064 _step‚238 & lr-AdamW/pg1‚0.0002017047595589703 & lr-AdamW/pg2‚0.0002017047595589703 trainer/global_step‚2949 % _timestamp‚1.7883977890926461e+09¹`ÉÌÆ & lr-AdamW/pg2‚0.0002017047595589703 trainer/global_step‚2949 % _timestamp‚1.7883977890926461e+09 _runtime‚2413.729555064 _step‚238 & lr-AdamW/pg1‚0.0002017047595589703î­é=2‚- _wandbruntime‚2413  _runtime‚24134s×wƒÁO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4154067369472 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59498459 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210585088 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚82939961137961 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4141935399345 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98221025 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22589354 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59518496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 trainer/global_step‚2949 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38702529 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39344437 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16755083 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚86988182955008 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚86988369957376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87774435621888 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22589356 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚82834302587904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚87774459561984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚82834284584960 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚78797860617216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚82834302587904 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚82834284584960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38702527 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚2413.730551615 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98200986 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16755612 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚86988369957376 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚92001015860224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7883977890934353e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86988182955008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4153898370048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127`ØåµÌ'„ÆO [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚82834284584960 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚78797860617216 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚82834302587904 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚82834284584960 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚38702527 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚2413.730551615 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚98200986 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚16755612 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚86988369957376 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚92001015860224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7883977890934353e+09 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚86988182955008 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4153898370048 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4154067369472 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚59498459 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210585088 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚82939961137961 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚531 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4141935399345 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚98221025 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚22589354 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚59518496 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 trainer/global_step‚2949 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚38702529 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39344437 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚16755083 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚86988182955008 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚86988369957376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚87774435621888 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22589356 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚82834302587904 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚87774459561984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944ïÓ_£X2…- _wandbruntime‚2414  _runtime‚2414”$ÆžÑ&†ËM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98233432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38715188 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚82861553949696 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚82861553949696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚78823803799552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 trainer/global_step‚2949 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚87017032473600 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚92031578854400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4143489685497 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚87016845457920 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚82967111324081 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚87017032473600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16760364 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4228206344192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474624 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚78823786760192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38715186 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98233432 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22596733 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4227997869568 _step‚240 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7883977897848935e+09 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4155460520960 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59538309 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39357095 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98253497 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59518246 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚87803372510208 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ODeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4155460520960 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚59538309 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39357095 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚98253497 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚59518246 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚87803372510208 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ODeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚98233432 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚38715188 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚82861553949696 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚82861553949696 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚78823803799552 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 trainer/global_step‚2949 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚87017032473600 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚92031578854400 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4143489685497 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚87016845457920 =7DeviceStatsMonitor.on_train_batch_end/num_allocæ0œÔ_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚82967111324081 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚87017032473600 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚16760364 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4228206344192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474624 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚78823786760192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚38715186 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚98233432 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22596733 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4227997869568 _step‚240 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7883977897848935e+09ðQ«ï&2ˆ- _wandbruntime‚2414  _runtime‚2414únÛCZ‰Ô %loss/total_loss‚50.40742954679075 trainer/global_step‚2949 /loss/FRt4/detection_loss‚1.6773539123435814 1loss/SRqqt3/detection_loss‚2.0111656715261175 epoch‚2 _step‚241 0loss/FRt1/assignment_loss‚4.2084759084545835 2loss/SRqqt3/assignment_loss‚1.7854235955375306 0loss/FBt3/assignment_loss‚0.3822457542022069 .loss/FBt2/detection_loss‚3.156406353820454 /loss/FBt4/detection_loss‚0.8520496330503263 2loss/SRqqt4/assignment_loss‚1.3266205988614492 .loss/FRt3/detection_loss‚2.786086926609278 1loss/SRqqt1/detection_loss‚1.9950149177477277 0loss/SRqqt2/detection_loss‚2.503281786586299 .loss/FBt3/detection_loss‚3.951846957206726 0loss/FRt3/assignment_loss‚3.3768993631005286 0loss/FBt1/assignment_loss‚0.4995286318153376 0loss/FRt4/assignment_loss‚1.5228068778912227 1loss/FBt4/assignment_loss‚0.21816113948693783 /loss/FRt1/detection_loss‚2.0604646157245248 /loss/FRt2/detection_loss‚2.8754410799592733 1loss/SRqqt4/detection_loss‚0.9955191286936612 % _timestamp‚1.7883977897870116e+09 0loss/FBt2/assignment_loss‚0.5408379025920195 _runtime‚2414.423781291 /loss/FRt2/assignment_loss‚4.542639199644327 2loss/SRqqt1/assignment_loss‚2.0376777510190833 2loss/SRqqt2/assignment_loss‚2.2646770933360765 .loss/FBt1/detection_loss‚2.836804747581482ì€Î_ŠÙ %loss/total_loss‚50.40742954679075 trainer/global_step‚2949 /loss/FRt4/detection_loss‚1.6773539123435814 1loss/SRqqt3/detection_loss‚2.0111656715261175 epoch‚2 _step‚241 0loss/FRt1/assignment_loss‚4.2084759084545835 2loss/SRqqt3/assignment_loss‚1.7854235955375306 0loss/FBt3/assignment_loss‚0.3822457542022069 .loss/FBt2/detection_loss‚3.156406353820454 /loss/FBt4/detection_loss‚0.8520496330503263 2loss/SRqqt4/assignment_loss‚1.3266205988614492 .loss/FRt3/detection_loss‚2.786086926609278 1loss/SRqqt1/detection_loss‚1.9950149177477277 0loss/SRqqt2/detection_loss‚2.503281786586299 .loss/FBt3/detection_loss‚3.951846957206726 0loss/FRt3/assignment_loss‚3.3768993631005286 0loss/FBt1/assignment_loss‚0.4995286318153376 0loss/FRt4/assignment_loss‚1.5228068778912227 1loss/FBt4/assignment_loss‚0.21816113948693783 /loss/FRt1/detection_loss‚2.0604646157245248 /loss/FRt2/detection_loss‚2.8754410799592733 1loss/SRqqt4/detection_loss‚0.9955191286936612 % _timestamp‚1.7883977897870116e+09 0loss/FBt2/assignment_loss‚0.5408379025920195 _runtime‚2414.423781291 /loss/FRt2/assignment_loss‚4.542639199644327 2loss/SRqqt1/assignment_loss‚2.0376777510190833 2loss/SRqqt2/assignment_loss‚2.2646770933360765 .loss/FBt1/detection_loss‚2.836804747581482ñÚ® á‹‚(:’ Þ‘ãÔ’Ù«® gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚129.949' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.989800000000002 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÇEpŒ‚(:å Þ‘ãÔΈ§±$ memory_percent‚91.46201610565186# proc.memory.rssMB‚ 2693.83984375 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 6090.0859375 disk.sda.in‚ 0.05078125 disk.drbd1716.in‚0) proc.memory.availableMB‚ 1398.86328125* proc.memory.percent‚16.441893577575684 network.sent‚ 6.357666e+06 disk.dm-0.in‚ 334.1484375 disk.drbd2266.in‚0! disk.nvme0c0n1.in‚ 334.1484375" disk.nvme0n1.out‚ 18752.8515625 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 9.36767578125 cpu‚25.328164956888255 disk.dm-1.in‚ 0.05078125 disk.drbd2266.out‚0 disk.nvme0n1.in‚ 334.1484375 disk.rbd0.in‚0 network.recv‚888694 disk.dm-1.out‚ 6090.0859375 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-0.out‚ 18752.8515625 disk.sda2.in‚0 disk.sda2.out‚0$ disk./.usageGB‚252.2152099609375 disk.sda3.in‚ 0.05078125 disk.sda.out‚ 6090.0859375* disk./.usagePercent‚16.926374363135178 disk.drbd2265.in‚0 disk.rbd1.in‚0$ disk.nvme0c0n1.out‚ 18752.8515625“‹z‘‚(:† í‘ãÔð©ú­ gpu.0.gpu‚7.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚137.103' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.4206 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„ÕhuŽ‚(:ê í‘ãÔ„¦â±# disk.nvme0n1.out‚18753.14453125 disk.sda2.out‚0 disk.dm-1.out‚ 6109.51171875 disk.drbd2080.out‚0 cpu‚25.326455370737023% disk./.usageGB‚252.21519470214844 disk.sda.in‚ 0.05078125 disk.dm-1.in‚ 0.05078125 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 network.recv‚894662 disk.nvme0n1.in‚ 334.1484375 disk.sda3.out‚ 6109.51171875# proc.memory.rssMB‚ 2693.83984375) disk./.usagePercent‚16.92637333910504 disk.rbd1.in‚0 disk.rbd1.out‚ 9.45361328125 disk.sda1.in‚0 disk.sda3.in‚ 0.05078125 proc.cpu.threads‚17 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0% disk.nvme0c0n1.out‚18753.14453125 disk.sda.out‚ 6109.51171875 disk.drbd2266.out‚0$ memory_percent‚91.46246910095215* proc.memory.percent‚16.441893577575684 network.sent‚ 6.465815e+06 disk.drbd1716.out‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 334.1484375 disk.dm-0.out‚18753.14453125( proc.memory.availableMB‚ 1398.7890625! disk.nvme0c0n1.in‚ 334.1484375!ýeÉ‚(:’ ü‘ãÔØ–¯® gpu.0.gpu‚32.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚124.287' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.857400000000002 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=(°.s‚(:è ü‘ãÔê—â°# proc.memory.rssMB‚ 2693.83984375 network.recv‚898957% disk./.usageGB‚252.21514511108398 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.rbd1.in‚0) proc.memory.availableMB‚ 1398.54296875 disk.drbd2265.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.05078125 disk.dm-0.out‚ 18753.6015625" disk.nvme0n1.out‚ 18753.6015625$ memory_percent‚91.46397113800049 disk.sda.in‚ 0.05078125 disk.sda1.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 334.1484375 disk.sda3.out‚ 6138.75390625 disk.nvme0n1.in‚ 334.1484375 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 9.51611328125! disk.nvme0c0n1.in‚ 334.1484375 cpu‚25.32489893860969 proc.cpu.threads‚17* disk./.usagePercent‚16.926370011007094 disk.sda2.in‚0 disk.dm-1.out‚ 6138.75390625 disk.drbd2080.in‚0 disk.sda3.in‚ 0.05078125$ disk.nvme0c0n1.out‚ 18753.6015625* proc.memory.percent‚16.441893577575684 network.sent‚ 6.471812e+06 disk.sda.out‚ 6138.75390625 disk.drbd2266.in‚0ÞÛú¯2‘- _wandbruntime‚2450  _runtime‚2450ÐÂ8È’Â ' lr-AdamW/pg1‚0.00020094776594645886 ' lr-AdamW/pg2‚0.00020094776594645886 trainer/global_step‚2999 $ _timestamp‚1.788397825833542e+09 _runtime‚2450.470442757 _step‚242eÎ@ŽÍ“Ç _step‚242 ' lr-AdamW/pg1‚0.00020094776594645886 ' lr-AdamW/pg2‚0.00020094776594645886 trainer/global_step‚2999 $ _timestamp‚1.788397825833542e+09 _runtime‚2450.470442757òBBQp2”- _wandbruntime‚2450  _runtime‚2450r³–Å'•¿O LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99826638 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4223738692096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚88421509516800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93517488164864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚84197752821760 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99846677 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚89220106973184 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _step‚243 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚524 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚84197770824704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17025297 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚93517720592896 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89220083033088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4223569692672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60491161 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88421322514432 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4223738692096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60491161 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚526 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487936 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4211407217753 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60511198 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚88421322514432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚2450.471343998 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚84306409912409 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39335479 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚88421509516800 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39335479 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39984027 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39983501 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22958206 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360#q \Ê'–ÄO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _runtime‚2450.471343998 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚84306409912409 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚39335479 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚88421509516800 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 IDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39335479 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚39984027 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚39983501 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚22958206 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 E8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚99826638 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4223738692096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚88421509516800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚93517488164864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚84197752821760 M@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚99846677 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚89220106973184 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _step‚243 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚524 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚84197770824704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17025297 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚93517720592896 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚89220083033088 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4223569692672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚60491161 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚88421322514432 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 IDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4223738692096 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚60491161 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚526 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487936 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4211407217753 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚60511198 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚88421322514432óæ>¨(2—- _wandbruntime‚2451  _runtime‚2451ãC*È¢˜ËM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚84225022186496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4298816119296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4212961503905 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4224962830848 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4212796382297 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚89249019921408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60531011 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocat‹"‡4/ion.large_pool.freed‚39348136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242900480 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99879149 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39348138 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚80120945876992 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39348138 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22965583 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚244 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39996280 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4225131843584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60510948 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84333742259289 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4225131843584 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚84225040189440 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99879149 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚88449985017344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60531011 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4224962830848 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93548055001088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89248995981312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39996794 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17030699 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84225022186496 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84333924420257 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218960384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _runtime‚2451.172631401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99859084 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22965581 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚88450172033024 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99859084 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60510948 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 trainer/global_step‚2999 % _timestamp‚1.7883978265352743e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93547812100608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065Ò{OFÖ&™ÐM OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17030699 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚84225022186496 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚84333924420257 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218960384 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _runtime‚2451.172631401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 G:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚99859084 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚22965581 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚88450172033024 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 C6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚99859084 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚60510948 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 trainer/global_step‚2999 % _timestamp‚1.7883978265352743e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚93547812100608 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚84225022186496 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4298816119296 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4212961503905 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4224962830848 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4212796382297 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚89249019921408 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚60531011 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚39348136 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242900480 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 K>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚99879149 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39348138 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚80120945876992 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39348138 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚22965583 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚244 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚39996280 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4225131843584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚60510948 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚84333742259289 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4225131843584 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚84225040189440 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 G:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚99879149 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚88449985017344 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚60531011 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4224962830848 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚93548055001088 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚89248995981312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚39996794 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160ôdÊäV2š- _wandbruntime‚2451  _runtime‚2451#Ý$1X›Ò /loss/FRt2/assignment_loss‚4.909825682640076 1loss/SRqqt4/assignment_loss‚1.134996097534895 .loss/FRt1/detection_loss‚2.404904623205463 trainer/global_step‚2999 _runtime‚2451.174043136 -loss/FBt1/detection_loss‚2.18243864774704 .loss/FBt2/detection_loss‚3.230712127685547 2loss/SRqqt2/assignment_loss‚2.1586693479464603 2loss/SRqqt3/assignment_loss‚1.4577652969956398 /loss/FRt3/detection_loss‚2.2818645346164703 /loss/FRt1/assignment_loss‚4.036598434050878 1loss/SRqqt1/detection_loss‚1.7187443029768061 0loss/SRqqt2/detection_loss‚2.363051876043662 1loss/SRqqt3/detection_loss‚1.7362806802988053 /loss/FBt3/detection_loss‚2.9523260252816335 epoch‚2 0loss/FRt3/assignment_loss‚3.6650863981246946 0loss/FBt1/assignment_loss‚0.6263127146230545 _step‚245 0loss/FBt2/assignment_loss‚0.6055047889115371 0loss/FBt3/assignment_loss‚0.4413526781967708 %loss/total_loss‚49.16763809505132 % _timestamp‚1.7883978265372047e+09 /loss/FBt4/detection_loss‚1.0282348140701303 0loss/FRt4/assignment_loss‚2.1685611204667525 2loss/SRqqt1/assignment_loss‚1.8037473841939333 1loss/FBt4/assignment_loss‚0.35748582321807143 /loss/FRt2/detection_loss‚2.9648539304733275 /loss/FRt4/detection_loss‚1.8300456109372052 1loss/SRqqt4/detection_loss‚1.1082751548124685[  ]œ× 0loss/FBt3/assignment_loss‚0.4413526781967708 %loss/total_loss‚49.16763809505132 % _timestamp‚1.7883978265372047e+09 /loss/FBt4/detection_loss‚1.0282348140701303 0loss/FRt4/assignment_loss‚2.1685611204667525 2loss/SRqqt1/assignment_loss‚1.8037473841939333 1loss/FBt4/assignment_loss‚0.35748582321807143 /loss/FRt2/detection_loss‚2.9648539304733275 /loss/FRt4/detection_loss‚1.8300456109372052 1loss/SRqqt4/detection_loss‚1.1082751548124685 /loss/FRt2/assignment_loss‚4.909825682640076 1loss/SRqqt4/assignment_loss‚1.134996097534895 .loss/FRt1/detection_loss‚2.404904623205463 trainer/global_step‚2999 _runtime‚2451.174043136 -loss/FBt1/detection_loss‚2.18243864774704 .loss/FBt2/detection_loss‚3.230712127685547 2loss/SRqqt2/assignment_loss‚2.1586693479464603 2loss/SRqqt3/assignment_loss‚1.4577652969956398 /loss/FRt3/detection_loss‚2.2818645346164703 /loss/FRt1/assignment_loss‚4.036598434050878 1loss/SRqqt1/detection_loss‚1.7187443029768061 0loss/SRqqt2/detection_loss‚2.363051876043662 1loss/SRqqt3/detection_loss‚1.7362806802988053 /loss/FBt3/detection_loss‚2.9523260252816335 epoch‚2 0loss/FRt3/assignment_loss‚3.6650863981246946 0loss/FBt1/assignment_loss‚0.6263127146230545 _step‚245 0loss/FBt2/assignment_loss‚0.6055047889115371õ8EOb‚(:’ ‹’ãÔðž±® gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚135.908' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.181599999999996 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0t¿òplž‚(:á ‹’ãÔº”®± network.sent‚ 6.478041e+06 disk.rbd0.out‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 334.1484375$ disk.nvme0c0n1.out‚ 18753.8515625# proc.memory.rssMB‚ 2693.83984375* disk./.usagePercent‚16.926373595112572$ disk./.usageGB‚252.2151985168457 disk.sda.in‚ 0.05078125 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 cpu‚25.323126199154167 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.sda3.in‚ 0.05078125" disk.nvme0n1.out‚ 18753.8515625 disk.sda.out‚ 6170.859375 disk.dm-0.out‚ 18753.8515625 disk.sda1.in‚0 disk.sda3.out‚ 6170.859375( proc.memory.availableMB‚ 1398.7734375 network.recv‚903742 disk.drbd1716.out‚0 disk.dm-1.out‚ 6170.859375! disk.nvme0c0n1.in‚ 334.1484375$ memory_percent‚91.46256446838379 disk.nvme0n1.in‚ 334.1484375 disk.dm-1.in‚ 0.05078125 disk.drbd2265.in‚0 disk.rbd1.in‚0* proc.memory.percent‚16.441893577575684 disk.drbd2080.out‚0 disk.rbd1.out‚ 9.51611328125ï.WK’Ÿ‚(:‡ š’ãÔ†§û­ gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚138.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.6658 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»[ˆß| ‚(:ñ š’ãÔ„ü° disk.sda2.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0) proc.memory.availableMB‚ 1398.74609375 cpu‚25.321596006907647 network.sent‚ 6.585719e+06% disk./.usageGB‚252.21514129638672 disk.sda.out‚ 6204.61328125 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.dm-1.out‚ 6204.61328125 disk.rbd0.out‚0 disk.dm-0.in‚ 334.1484375 disk.sda1.in‚0 disk.sda3.in‚ 0.05078125! disk.nvme0c0n1.in‚ 334.1484375' disk.nvme0c0n1.out‚18754.3291015625" disk.dm-0.out‚18754.3291015625) disk./.usagePercent‚16.92636975499956% disk.nvme0n1.out‚18754.3291015625 disk.sda2.out‚0 disk.drbd2080.out‚0$ memory_percent‚91.46273136138916 disk.sda1.out‚0 disk.sda3.out‚ 6204.61328125 disk.rbd1.out‚ 9.54833984375 disk.drbd2265.out‚0* proc.memory.percent‚16.441893577575684 proc.cpu.threads‚17 disk.nvme0n1.in‚ 334.1484375 disk.sda.in‚ 0.05078125 disk.dm-1.in‚ 0.05078125 disk.drbd2266.in‚0# proc.memory.rssMB‚ 2693.83984375 network.recv‚909044 disk.rbd1.in‚0 disk.drbd2266.out‚06z¿2¡- _wandbruntime‚2487  _runtime‚2487éØ É¢Ã _step‚246 ' lr-AdamW/pg1‚0.00020017129656473833 ' lr-AdamW/pg2‚0.00020017129656473833 trainer/global_step‚3049 % _timestamp‚1.7883978628968282e+09 _runtime‚2487.533718906nÊOÎ£È ' lr-AdamW/pg2‚0.00020017129656473833 trainer/global_step‚3049 % _timestamp‚1.7883978628968282e+09 _runtime‚2487.533718906 _step‚246 ' lr-AdamW/pg1‚0.00020017129656473833öªã¿`2¤- _wandbruntime‚2487  _runtime‚2487E Û !¥ÅO O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚85673040847617 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚61483863 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚40623783 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 % _timestamp‚1.7883978628975725e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4368290858496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4293410014720 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚61503900 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚85561239061504 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4368505637888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23327056 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚89854462073856 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39968429 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61483863 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238719488 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚81392178850816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61503900 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚89854462073856 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39968427 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101472329 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚81392161811456 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 ZLDeviceStatsMonitor.on_train_batch_start/inacti£ ¬V+ve_split_bytes.large_pool.peak‚ 104436224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚247 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚85561221058560 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚90665754384384 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚85561239061504 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4368505637888 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23327056 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚89854462073856 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚39968429 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚61483863 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238719488 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚81392178850816 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚61503900 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚89854462073856 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚39968427 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 101472329 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚81392161811456 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _step‚247 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚85561221058560 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚90665754384384 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 % _timestamp‚1.7883978628975725e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4368290858496 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1÷nùÆ2§- _wandbruntime‚2488  _runtime‚2488#(žÓ&¨ÍM ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚85588508426240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4282268200705 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61523713 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17302254 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90694643392512 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4294803166208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17302766 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101504801 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚85588508426240 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4294803166208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23334433 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 _step‚248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3049 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 $ _timestamp‚1.788397863594851e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚85700373194497 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61503650 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚89883124576768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4282433322313 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39981088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23334431 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101484736 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39981086 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101504801 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39981088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚89883311592448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40637199 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40636685 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4294634153472 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101484736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206377472 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚81418122033152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0¥mzØ&©ÒM J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚61503650 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚89883124576768 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4282433322313 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚39981088 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23334431 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 101484736 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚39981086 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 101504801 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚39981088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚89883311592448 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚40637199 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚40636685 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4294634153472 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 101484736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317568 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206377472 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚81418122033152 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚85588508426240 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4282268200705 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚61523713 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17302254 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚90694643392512 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4294803166208 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17302766 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 101504801 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚85588508426240 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4294803166208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23334433 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 _step‚248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚3049 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 $ _timestamp‚1.788397863594851e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚85700373194497 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199øì J¸2ª- _wandbruntime‚2488  _runtime‚2488³²bÅ«Ñ 1loss/SRqqt2/detection_loss‚2.6423437856137753 1loss/SRqqt4/detection_loss‚1.1271410425720008 trainer/global_step‚3049 % _timestamp‚1.7883978635965004e+09 .loss/FRt2/detection_loss‚2.933397127315402 1loss/SRqqt1/detection_loss‚2.2398412352384525 /loss/FBt4/detection_loss‚0.6829872412380593 $loss/total_loss‚49.9350004639665 /loss/FRt2/assignment_loss‚4.319110631942749 1loss/SRqqt4/assignment_loss‚1.298282546814585 1loss/FBt4/assignment_loss‚0.18377822125504975 .loss/FRt3/detection_loss‚2.586239232044471 epoch‚2 _step‚249 2loss/SRqqt1/assignment_loss‚1.4961445286523465 1loss/FBt1/assignment_loss‚0.34489207789301873 1loss/SRqqt3/detection_loss‚1.7471268608101778 .loss/FBt1/detection_loss‚4.160095381736755 _runtime‚2488.233418942 /loss/FRt1/assignment_loss‚4.642280617485875 0loss/FRt3/assignment_loss‚2.7231393958392895 0loss/FBt2/assignment_loss‚0.60955632879uÎ’90444 .loss/FBt3/detection_loss‚3.035320907831192 2loss/SRqqt3/assignment_loss‚1.3520421916618943 /loss/FRt1/detection_loss‚2.2734998477541883 .loss/FBt2/detection_loss‚3.059395730495453 0loss/FRt4/assignment_loss‚1.8822446167469025 .loss/FRt4/detection_loss‚1.740043114732813 2loss/SRqqt2/assignment_loss‚2.4659615501295775 0loss/FBt3/assignment_loss‚0.3901362493634224!"»é\¬Ö 2loss/SRqqt2/assignment_loss‚2.4659615501295775 0loss/FBt3/assignment_loss‚0.3901362493634224 1loss/SRqqt2/detection_loss‚2.6423437856137753 1loss/SRqqt4/detection_loss‚1.1271410425720008 trainer/global_step‚3049 % _timestamp‚1.7883978635965004e+09 .loss/FRt2/detection_loss‚2.933397127315402 1loss/SRqqt1/detection_loss‚2.2398412352384525 /loss/FBt4/detection_loss‚0.6829872412380593 $loss/total_loss‚49.9350004639665 /loss/FRt2/assignment_loss‚4.319110631942749 1loss/SRqqt4/assignment_loss‚1.298282546814585 1loss/FBt4/assignment_loss‚0.18377822125504975 .loss/FRt3/detection_loss‚2.586239232044471 epoch‚2 _step‚249 2loss/SRqqt1/assignment_loss‚1.4961445286523465 1loss/FBt1/assignment_loss‚0.34489207789301873 1loss/SRqqt3/detection_loss‚1.7471268608101778 .loss/FBt1/detection_loss‚4.160095381736755 _runtime‚2488.233418942 /loss/FRt1/assignment_loss‚4.642280617485875 0loss/FRt3/assignment_loss‚2.7231393958392895 0loss/FBt2/assignment_loss‚0.6095563287990444 .loss/FBt3/detection_loss‚3.035320907831192 2loss/SRqqt3/assignment_loss‚1.3520421916618943 /loss/FRt1/detection_loss‚2.2734998477541883 .loss/FBt2/detection_loss‚3.059395730495453 0loss/FRt4/assignment_loss‚1.8822446167469025 .loss/FRt4/detection_loss‚1.740043114732813ùã}v’­‚(:‡ ©’ãÔÄ뇮 gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚128.353' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.6706 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01®Êö{®‚(:ð ©’ãÔ¢¶Œ± disk.sda.out‚ 6237.16796875 disk.sda2.in‚0" disk.dm-0.out‚18754.5595703125 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 334.1484375 disk.drbd1716.in‚0 disk.rbd1.out‚ 9.63427734375% disk.nvme0n1.out‚18754.5595703125 disk.sda2.out‚0 disk.dm-1.out‚ 6237.16796875$ memory_percent‚91.46304130554199$ disk./.usageGB‚252.2157211303711 disk.drbd2266.out‚0 disk.rbd1.in‚0* proc.memory.percent‚16.441893577575684 disk.sda.in‚ 0.05078125* disk./.usagePercent‚16.926408668144767 disk.drbd2080.out‚0 disk.dm-0.in‚ 334.1484375 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 334.1484375' disk.nvme0c0n1.out‚18754.5595703125 disk.drbd1716.out‚0 network.sent‚ 6.591636e+06 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 6237.16796875# proc.memory.rssMB‚ 2693.83984375 proc.cpu.threads‚17 network.recv‚913173 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.05078125( proc.memory.availableMB‚ 1398.6953125 disk.dm-1.in‚ 0.05078125 cpu‚25.319873787608874R„·Ð¯‚(:’ ¸’ãԶ픮 gpu.0.gpu‚22.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚111.772' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.354400000000002 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢fG~°‚(:ó ¸’ãÔü÷º° proc.cpu.threads‚17 disk.dm-0.in‚ 334.1484375 disk.sda.in‚ 0.05078125* proc.memory.percent‚16.441893577575684 cpu‚25.31838581115253 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0# disk.dm-0.out‚18759.85595703125& disk.nvme0n1.out‚18759.85595703125) proc.memory.availableMB‚ 1398.41796875 network.sent‚ 6.700044e+06 disk.dm-1.in‚ 0.05078125 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 334.1484375 network.recv‚919599 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚ 0.05078125$ memory_percent‚91.46473407745361* disk./.usagePercent‚16.837481123028457 disk.drbd1716.in‚0 disk.dm-1.out‚ 6273.99609375 disk.rbd1.out‚ 9.69677734375 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.sda.out‚ 6273.99609375# proc.memory.rssMB‚ 2693.83984375( disk.nvme0c0n1.out‚18759.85595703125 disk.sda2.in‚0! disk.nvme0c0n1.in‚ 334.1484375$ disk./.usageGB‚250.8906364440918 disk.drbd2080.out‚0 disk.sda3.out‚ 6273.99609375 disk.rbd1.in‚0|©Lw’±‚(:‡ Ç’ãÔ±¥·® gpu.0.gpu‚19.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚116.781' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.3562 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'H»÷y²‚(:î Ç’ãÔÑ”º±& disk.nvme0n1.out‚18760.30908203125 disk.sda.out‚ 6302.640625 disk.drbd2266.in‚0 disk.drbd2265.in‚0 proc.cpu.threads‚17 disk.drbd2265.out‚0$ memory_percent‚91.46783351898193 disk.dm-0.in‚ 334.1484375 disk.sda1.in‚0 disk.sda3.out‚ 6302.640625 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0) proc.memory.availableMB‚ 1397.91015625# disk.dm-0.out‚18760.30908203125 disk.sda1.out‚0 disk.sda.in‚ 0.05078125 disk.dm-1.out‚ 6302.640625 disk.drbd2080.out‚0* proc.memory.percent‚16.441893577575684 network.recv‚924122 disk.sda2.out‚0 network.sent‚ 6.70642e+06 disk.sda2.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2693.83984375* disk./.usagePercent‚16.837473442802427 disk.rbd1.out‚ 9.69677734375 disk.nvme0n1.in‚ 334.1484375 disk.drbd2266.out‚0 cpu‚25.316685063990132 disk.rbd1.in‚0( disk.nvme0c0n1.out‚18760.30908203125 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.05078125% disk./.usageGB‚250.89052200317383 disk.sda3.in‚ 0.05078125! disk.nvme0c0n1.in‚ 334.1484375* ùù2³- _wandbruntime‚2524  _runtime‚2524ÒûÃÉ´Ã _runtime‚2524.589327889 _step‚250 ' lr-AdamW/pg1‚0.00019937551322886406 ' lr-AdamW/pg2‚0.00019937551322886406 trainer/global_step‚3099 % _timestamp‚1.7883978999524324e+09€Ò›ÎµÈ _step‚250 ' lr-AdamW/pg1‚0.00019937551322886406 ' lr-AdamW/pg2‚0.00019937551322886406 trainer/global_step‚3099 % _timestamp‚1.7883978999524324e+09 _runtime‚2524.589327889ú¶•R&2¶- _wandbruntime‚2524  _runtime‚2524d êôË'·ÅO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚82689320928256 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚96550831725568 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96550607686144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚91287788635648 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4362912337920 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4350515975969 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚92111401795584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103077942 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚91287788635648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7883978999531775e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚86924689295360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82689337967616 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200099328 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224039424 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4363081337344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4439229830656 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚86924707298304 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚91287601633280 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62476565 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103097981 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41262193 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23695906 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚87039853943585 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚86924689295360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4362912337920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62496602 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚87039671782825ë0ûÐ'¸ÊO XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚86924689295360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4362912337920 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚62496602 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚87039671782825 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚82689320928256 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚96550831725568 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚96550607686144 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚91287788635648 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4362912337920 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4350515975969 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚92111401795584 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 103077942 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚91287788635648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7883978999531775e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚86924689295360 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚82689337967616 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚528 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200099328 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 224039424 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4363081337344 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4439229830656 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚86924707298304 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚91287601633280 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚62476565 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 103097981 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚41262193 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚23695906 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚87039853943585 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217ûƒ2¿2¹- _wandbruntime‚2525  _runtime‚2525…Z¥¥§ºÏM ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚86951976663040 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 trainer/global_step‚3099 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86951976663040 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41275367 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚252 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40614036 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17572084 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4364474488832 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91316451151872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4351740019113 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103110388 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚518 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚91316451151872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103130453 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚86951958660096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _runtime‚2525.289758312 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4364305476096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40614038 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41274849 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚516 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96581169515008 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4364474488832 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40614036 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚91316264136192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚92140290803712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚86951958660096 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103130453 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23703283 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4440646296576 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23703281 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚82715264110592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2}M6Ú&»ÔM VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚86951958660096 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 103130453 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚23703283 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4440646296576 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚23703281 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚82715264110592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚86951976663040 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 trainer/global_step‚3099 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚86951976663040 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41275367 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚252 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚40614036 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17572084 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4364474488832 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚91316451151872 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4351740019113 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 103110388 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚518 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚91316451151872 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 103130453 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚86951958660096 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _runtime‚2525.289758312 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4364305476096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚40614038 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41274849 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚516 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚96581169515008 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4364474488832 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚40614036 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚91316264136192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚92140290803712 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160ü«P2¼- _wandbruntime‚2525  _runtime‚2525ÚzJ[½Õ 2loss/SRqqt4/assignment_loss‚1.1334644810902694 .loss/FRt1/detection_loss‚2.259322307326577 epoch‚2 /loss/FRt3/assignment_loss‚3.227901134735499 0loss/FRt4/assignment_loss‚1.6694044482474233 1loss/FBt1/assignment_loss‚0.31338871518770856 /loss/FBt4/detection_loss‚0.7380393545679507 /loss/FRt2/assignment_loss‚5.269164553395024 1loss/FBt4/assignment_loss‚0.22793552889554358 .loss/FBt2/detection_loss‚3.241923596177782 2loss/SRqqt1/assignment_loss‚2.0529447200623427 1loss/FBt3/assignment_loss‚0.42384727691891244 1loss/SRqqt3/detection_loss‚1.9263732692469722 .loss/FBt1/detection_loss‚2.556297691663106 1loss/SRqqt2/assignment_loss‚2.005753279537768 1loss/SRqqt4/detection_loss‚1.0709595363291482 trainer/global_step‚3099 0loss/FBt2/assignment_loss‚0.9093071955888133 /loss/FRt2/detection_loss‚3.0506076327076665 1loss/SRqqt1/detection_loss‚1.7507218014110217 _step‚253 /loss/FRt3/detection_loss‚2.5242540545952625 1loss/SRqqt2/detection_loss‚2.1297337026209444 /loss/FBt3/detection_loss‚2.9474262457627516 %loss/total_loss‚49.26552616444632 _runtime‚2525.291402656 .loss/FRt4/detection_loss‚1.977971402599531 % _timestamp‚1.7883979006545548e+09 0loss/FRt1/assignment_loss‚3.8636230623180214 2loss/SRqqt3/assignment_loss‚1.9951611734602763Rh–y`¾Ú 2loss/SRqqt1/assignment_loss‚2.0529447200623427 1loss/FBt3/assignment_loss‚0.42384727691891244 1loss/SRqqt3/detection_loss‚1.9263732692469722 .loss/FBt1/detection_loss‚2.556297691663106 1loss/SRqqt2/assignment_loss‚2.005753279537768 1loss/SRqqt4/detection_loss‚1.0709595363291482 trainer/global_step‚3099 0loss/FBt2/assignment_loss‚0.9093071955888133 /loss/FRt2/detection_loss‚3.0506076327076665 1loss/SRqqt1/detection_loss‚1.7507218014110217 _step‚253 /loss/FRt3/detection_loss‚2.5242540545952625 1loss/SRqqt2/detection_loss‚2.1297337026209444 /loss/FBt3/detection_loss‚2.9474262457627516 %loss/total_loss‚49.26552616444632 _runtime‚2525.291402656 .loss/FRt4/detection_loss‚1.977971402599531 % _timestamp‚1.7883979006545548e+09 0loss/FRt1/assignment_loss‚3.8636230623180214 2loss/SRqqt3/assignment_loss‚1.9951611734602763 2loss/SRqqt4/assignment_loss‚1.1334644810902694 .loss/FRt1/detection_loss‚2.259322307326577 epoch‚2 /loss/FRt3/assignment_loss‚3.227901134735499 0loss/FRt4/assignment_loss‚1.6694044482474233 1loss/FBt1/assignment_loss‚0.31338871518770856 /loss/FBt4/detection_loss‚0.7380393545679507 /loss/FRt2/assignment_loss‚5.269164553395024 1loss/FBt4/assignment_loss‚0.22793552889554358 .loss/FBt2/detection_loss‚3.241923596177782ý-—’¿‚(:‡ Ö’ãÔ›ò”® gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚111.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3284 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á×tÚxÀ‚(:í Ö’ãÔž°¿° disk.sda1.in‚0 disk.sda3.out‚ 6333.1953125! disk.nvme0c0n1.in‚ 334.1484375 disk.nvme0n1.in‚ 334.1484375 disk.sda.in‚ 0.05078125 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.sda3.in‚ 0.05078125# disk.dm-0.out‚18760.74951171875* proc.memory.percent‚16.441893577575684 cpu‚25.315142225042514 disk.drbd2080.out‚0# proc.memory.rssMB‚ 2693.83984375 proc.cpu.threads‚17 network.sent‚ 6.711779e+06 network.recv‚927420% disk./.usageGB‚250.89046478271484 disk.rbd1.out‚ 9.72900390625 disk.dm-0.in‚ 334.1484375 disk.rbd1.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 6333.1953125 disk.drbd2265.in‚0& disk.nvme0n1.out‚18760.74951171875* disk./.usagePercent‚16.837469602689414 disk.sda1.out‚0( disk.nvme0c0n1.out‚18760.74951171875 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 0.05078125$ proc.memory.availableMB‚1398.125 disk.sda.out‚ 6333.1953125 disk.drbd1716.in‚0 disk.drbd2265.out‚0$ memory_percent‚91.46652221679688›Ò Ý’Á‚(:‡ å’ãÔ㈬® gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚123.123' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.6246 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0üþþdv‚(:ë å’ãÔÙ§ê±# memory_percent‚91.4665937423706 disk.drbd2266.in‚0# disk.dm-0.out‚18760.96435546875 disk.sda1.in‚0 disk.sda3.out‚ 6364.40625* proc.memory.percent‚16.441893577575684 disk.sda.in‚ 0.05078125 network.sent‚ 6.820082e+06 disk.drbd2265.out‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚18760.96435546875 proc.cpu.threads‚17 disk.drbd2266.out‚0# proc.memory.rssMB‚ 2693.83984375 disk.rbd0.in‚0 disk.rbd0.out‚0 network.recv‚933538* disk./.usagePercent‚16.837469602689414 disk.dm-1.out‚ 6364.40625 disk.sda1.out‚0 disk.sda3.in‚ 0.05078125 disk.sda2.out‚0) proc.memory.availableMB‚ 1398.11328125 cpu‚25.313667569731887 disk.drbd2080.out‚0 disk.nvme0n1.in‚ 334.1484375& disk.nvme0n1.out‚18760.96435546875 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.05078125 disk.drbd2080.in‚0 disk.sda.out‚ 6364.40625% disk./.usageGB‚250.89046478271484 disk.drbd2265.in‚0 disk.dm-0.in‚ 334.1484375 disk.rbd1.out‚ 9.72900390625! disk.nvme0c0n1.in‚ 334.1484375 disk.sda2.in‚0ôðQ©2Ã- _wandbruntime‚2561  _runtime‚2561yaÁ,ÈÄ _step‚254 ' lr-AdamW/pg1‚0.00019856058177889052 ' lr-AdamW/pg2‚0.00019856058177889052 trainer/global_step‚3149 $ _timestamp‚1.788397936896547e+09 _runtime‚2561.5334522089,ÝÍÅÇ _step‚254 ' lr-AdamW/pg1‚0.00019856058177889052 ' lr-AdamW/pg2‚0.00019856058177889052 trainer/global_step‚3149 $ _timestamp‚1.788397936896547e+09 _runtime‚2561.533452208þhiúv2Æ- _wandbruntime‚2561  _runtime‚2561†Û DË'ÇÅO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104703594 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63489304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚83986497084416 _runtime‚2561.534391751 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98067374759936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41234327 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4419987794377 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚88288175535104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚92720741192704 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63489304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 % _timestamp‚1.7883979368972948e+09 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92720741192704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17837223 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63469267 trainer/global_step‚3149 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚92720928195072 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4510325553152 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93557025266688 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24064754 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4510108676608 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚255 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4419822672977 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63469267 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚98067133943296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚884064848787934 N›ÌÈÊO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚88288175535104 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚92720741192704 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚63489304 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 % _timestamp‚1.7883979368972948e+09 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚92720741192704 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚17837223 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚63469267 trainer/global_step‚3149 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚92720928195072 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4510325553152 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚93557025266688 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24064754 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4510108676608 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚255 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4419822672977 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚63469267 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚98067133943296 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚88406484878793 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 104703594 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚63489304 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚83986497084416 _runtime‚2561.534391751 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚98067374759936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚41234327 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4419987794377 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248ÿýÙ×û2É- _wandbruntime‚2562  _runtime‚2562tB¤Õ&ÊÏM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41246988 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104756105 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17843019 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7883979375885499e+09 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚92749403695616 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88433635064913 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚88315426896896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚92749590711296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4433976798720 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98097463238656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚88315444899840 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚88315444899840 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚84012440266752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236609024 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4511737692672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41915152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63489054 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚92749590711296 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4421211837521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚515 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚98097699847680 _step‚256 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17842506 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63509117 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41246986 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4434145811456 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41914637 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24072131 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚513 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41246988 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2562.225962004 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104736040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63489054 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚3149 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63509117 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104756105 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104736040 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84012423227392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4433976798720 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41246986 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚88315426896896 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4434145811456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93585938214912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608Gkó”Ú&ËÔM E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4434145811456 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚41914637 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24072131 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚513 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41246988 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚2562.225962004 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 104736040 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚63489054 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 trainer/global_step‚3149 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚63509117 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 104756105 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 104736040 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚84012423227392 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4433976798720 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚41246986 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚88315426896896 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4434145811456 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚93585938214912 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚41246988 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 104756105 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚17843019 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7883979375885499e+09 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚92749403695616 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚88433635064913 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚88315426896896 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚92749590711296 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4433976798720 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚98097463238656 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚88315444899840 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚88315444899840 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚84012440266752 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236609024 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4511737692672 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚41915152 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚63489054 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚92749590711296 O8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4421211837521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚515 [HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚98097699847680 _step‚256 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚17842506 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚63509117 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚41246986 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4502254983168 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚94154067754496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚89773115814001 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚99583956763136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚259 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚89651643771904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4489294491385 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4489459612785 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 trainer/global_step‚3199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106329246 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚95002696617984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18109614 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41867277 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106349285 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4581260145152 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚89651625768960 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4502254983168 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95002672677888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4581043268608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4502423982592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚94153880752128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚99583715946496 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚85283639161856 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 % _timestamp‚1.7883979737939622e+09 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚85283656201216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4502254983168 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚94154067754496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816640 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚89773115814001 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ]JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚99583956763136 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚259 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚89651643771904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4489294491385 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4489459612785 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 trainer/global_step‚3199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 106329246 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚95002696617984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18109614 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚41867277 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 106349285 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4581260145152 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚89651625768960 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4502254983168 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚95002672677888 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4581043268608 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4502423982592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚94153880752128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 YFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚99583715946496 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚85283639161856 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 % _timestamp‚1.7883979737939622e+09 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876544 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚85283656201216 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚89678913136640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42556224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚94182730270720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24440983 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64481756 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4503648121344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106381757 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚99614050969600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4490683655929 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64501819 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64481756 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚89800448161089 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4582465343488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18115767 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206377472 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4503648121344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 trainer/global_step‚3199 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.aä“&^ll.allocated‚94182730270720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚94182543255040 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚85309599383552 _step‚260 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106361692 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24440981 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95031609566208 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4490848777537 _runtime‚2599.126250751ö€Ú&ÝÔM FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 106381757 WDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚99614050969600 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4490683655929 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚64501819 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚64481756 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚89800448161089 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4582465343488 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18115767 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206377472 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4503648121344 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 trainer/global_step‚3199 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚94182730270720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚94182543255040 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚85309599383552 _step‚260 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 106361692 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24440981 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚95031609566208 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4490848777537 _runtime‚2599.126250751 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚89678895133696 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚85309582344192 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 106381757 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4503817134080 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚89800266000121 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚41879938 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚89678913136640 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚42556224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚94182730270720 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24440983 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚64481756 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4503648121344 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096„j/2Þ- _wandbruntime‚2599  _runtime‚2599• W UßÏ /loss/FRt1/assignment_loss‚4.354630799727007 0loss/FBt2/assignment_loss‚0.4574973454233259 epoch‚2 _runtime‚2599.127173462 /loss/FRt4/assignment_loss‚2.078503123549528 .loss/FRt1/detection_loss‚2.208319784294475 1loss/SRqqt3/detection_loss‚1.9968667687201986 %loss/total_loss‚49.81888935299891 1loss/FBt4/assignment_loss‚0.16850519979912673 .loss/FRt3/detection_loss‚2.463567512565189 0loss/SRqqt1/detection_loss‚1.907912622798573 1loss/SRqqt4/detection_loss‚1.3897864113025593 2loss/SRqqt3/assignment_loss‚1.4977103398770701 2loss/SRqqt4/assignment_loss‚1.3876825786675469 .loss/FRt2/detection_loss‚3.274752147819685 _step‚261 0loss/FBt3/assignment_loss‚0.6077146291330726 /loss/FBt2/detection_loss‚3.0950431406497954 /loss/FBt3/detection_loss‚3.2080947245870317 ,loss/FBt4/detection_loss‚0.7579925885384 /loss/FRt2/assignment_loss‚4.362233452174975 2loss/SRqqt2/assignment_loss‚2.1393402209205012 /loss/FRt4/detection_loss‚2.2473588996501856 0loss/SRqqt2/detection_loss‚2.430057523712035 trainer/global_step‚3199 % _timestamp‚1.7883979744903853e+09 0loss/FRt3/assignment_loss‚3.4506471541192796 2loss/SRqqt1/assignment_loss‚1.7887664463032376 0loss/FBt1/assignment_loss‚0.5251767168703833 /loss/FBt1/detection_loss‚2.0207292217957344ÁÓ!ZàÔ 1loss/FBt4/assignment_loss‚0.16850519979912673 .loss/FRt3/detection_loss‚2.463567512565189 0loss/SRqqt1/detection_loss‚1.907912622798573 1loss/SRqqt4/detection_loss‚1.3897864113025593 2loss/SRqqt3/assignment_loss‚1.4977103398770701 2loss/SRqqt4/assignment_loss‚1.3876825786675469 .loss/FRt2/detection_loss‚3.274752147819685 _step‚261 0loss/FBt3/assignment_loss‚0.6077146291330726 /loss/FBt2/detection_loss‚3.0950431406497954 /loss/FBt3/detection_loss‚3.2080947245870317 ,loss/FBt4/detection_loss‚0.7579925885384 /loss/FRt2/assignment_loss‚4.362233452174975 2loss/SRqqt2/assignment_loss‚2.1393402209205012 /loss/FRt4/detection_loss‚2.2473588996501856 0loss/SRqqt2/detection_loss‚2.430057523712035 trainer/global_step‚3199 % _timestamp‚1.7883979744903853e+09 0loss/FRt3/assignment_loss‚3.4506471541192796 2loss/SRqqt1/assignment_loss‚1.7887664463032376 0loss/FBt1/assignment_loss‚0.5251767168703833 /loss/FBt1/detection_loss‚2.0207292217957344 /loss/FRt1/assignment_loss‚4.354630799727007 0loss/FBt2/assignment_loss‚0.4574973454233259 epoch‚2 _runtime‚2599.127173462 /loss/FRt4/assignment_loss‚2.078503123549528 .loss/FRt1/detection_loss‚2.208319784294475 1loss/SRqqt3/detection_loss‚1.9968667687201986 %loss/total_loss‚49.81888935299891…u›©á‚(:… ¡“ãÔÛ¬œ® gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚123.51' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.702 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%,©Öpâ‚(:å ¡“ãÔ‚Ü„± disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.05078125 disk.drbd2265.in‚0 disk.dm-0.in‚ 334.1484375 disk.sda.out‚ 6498.1875 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.05078125# proc.memory.rssMB‚ 2693.83984375 cpu‚25.3075844382963 disk.rbd1.out‚ 9.90966796875( disk.nvme0c0n1.out‚18765.07080078125 disk.nvme0n1.in‚ 334.1484375 disk.drbd2266.in‚0# disk.dm-0.out‚18765.07080078125) proc.memory.availableMB‚ 1398.24609375$ disk./.usageGB‚249.5637969970703 disk.drbd1716.out‚0* proc.memory.percent‚16.441893577575684 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 6498.1875$ memory_percent‚91.46578311920166 network.recv‚951812 disk.drbd2080.in‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 334.1484375& disk.nvme0n1.out‚18765.07080078125* disk./.usagePercent‚16.748435814446395 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.05078125 disk.sda3.out‚ 6498.1875 network.sent‚ 6.94514e+06 disk.sda2.out‚0 disk.drbd2265.out‚0 <ªY’ã‚(:‡ °“ãԧѮ gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚136.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.3466 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–D†iä‚(:Þ °“ãÔé혰% disk.nvme0c0n1.out‚18765.31640625 disk.sda.out‚ 6527.9375 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 network.sent‚ 7.053513e+06 disk.sda3.in‚ 0.05078125 disk.sda3.out‚ 6527.9375 disk.drbd2266.out‚0 disk.dm-0.in‚ 334.1484375' proc.memory.availableMB‚ 1398.453125 proc.cpu.threads‚17! disk.nvme0c0n1.in‚ 334.1484375 disk.sda2.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 334.1484375 disk.sda.in‚ 0.05078125 disk.dm-1.in‚ 0.05078125# proc.memory.rssMB‚ 2693.83984375* proc.memory.percent‚16.441893577575684 disk.rbd1.out‚ 9.90966796875 disk.sda2.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0$ memory_percent‚91.46451950073242* disk./.usagePercent‚16.748434534408723# disk.nvme0n1.out‚18765.31640625 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.dm-0.out‚18765.31640625 cpu‚25.306143219373272 network.recv‚958056 disk.drbd1716.out‚0 disk.dm-1.out‚ 6527.9375 disk.rbd0.in‚0% disk./.usageGB‚249.56377792358398 disk.rbd1.in‚0žï‡T2å- _wandbruntime‚2635  _runtime‚2635¾Ý"üÉæà ' lr-AdamW/pg1‚0.00019687395781366236 ' lr-AdamW/pg2‚0.00019687395781366236 trainer/global_step‚3249 % _timestamp‚1.7883980109225276e+09 _runtime‚2635.559392071 _step‚26297UxÎçÈ % _timestamp‚1.7883980109225276e+09 _runtime‚2635.559392071 _step‚262 ' lr-AdamW/pg1‚0.00019687395781366236 ' lr-AdamW/pg2‚0.00019687395781366236 trainer/global_step‚3249†Ë*2è- _wandbruntime‚2635  _runtime‚2635iýœ¡Í'éÇO \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚91015112008704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚101100233084928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214779392 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42500229 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42500227 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 _runtime‚2635.560401837 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚86580815318016 % _timestamp‚1.7883980109232485e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65474708 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚101100471804416 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4651912995840 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24802454 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42500229 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚95587207313920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43183317 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚86580798278656 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚91015094005760 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚91015094005760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238719488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚96448320089088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚91139564588449 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107954898 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚95587020311552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4572095305216 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460´WK˜þêÌO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18380863 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚96448344029184 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4558766309793 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚42500229 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚42500227 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 _runtime‚2635.560401837 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚86580815318016 % _timestamp‚1.7883980109232485e+09 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚65474708 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚101100471804416 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4651912995840 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚24802454 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚42500229 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚95587207313920 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚43183317 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚86580798278656 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚91015094005760 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [HDekLËÔviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚91015094005760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238719488 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚96448320089088 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚91139564588449 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 107954898 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚95587020311552 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4572095305216 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚91015112008704 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚101100233084928 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214779392 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚91166896935329 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚96477256977408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7883980116135383e+09 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4560320595945 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 _step‚264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42512886 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65494521 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚ 2636.25131476 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24809831 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚3249 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43196422 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108007409 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚470 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚86606741460992 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24809833 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65474458 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4560155474337 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107987344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42512888 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4573319443968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4653541088256 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108007409 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚95615869830144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43195950 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65494521 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚86606758500352 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚91042381373440 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416¤x„Û&íÕM A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4573319443968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4573488456704 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚42512888 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18386589 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚101130798065664 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚95615869830144 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚91166896935329 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚96477256977408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7883980116135383e+09 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4560320595945 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 _step‚264 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚42512886 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚65494521 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474624 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚ 2636.25131476 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚24809831 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 trainer/global_step‚3249 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43196422 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 108007409 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚470 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚86606741460992 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚24809833 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚65474458 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4560155474337 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 107987344 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚42512888 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4573319443968 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4653541088256 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 108007409 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚95615869830144 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43195950 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚65494521 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚86606758500352 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚91042381373440 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ˆ„éY‰2î- _wandbruntime‚2636  _runtime‚2636¿RðŠYïÓ %loss/total_loss‚48.43360711713672 /loss/FRt1/assignment_loss‚4.459208913569181 2loss/SRqqt1/assignment_loss‚2.0273797703094973 1loss/FBt2/assignment_loss‚0.34641309497806105 .loss/FRt1/detection_loss‚2.099075326942048 1loss/SRqqt4/detection_loss‚1.0292234779580642 2loss/SRqqt3/assignment_loss‚1.4262653368596847 /loss/FBt2/detection_loss‚3.7477521672844887 /loss/FRt2/assignment_loss‚4.305407457053661 0loss/FRt4/assignment_loss‚1.6875236239589628 /loss/FBt4/detection_loss‚0.7841083074883101 2loss/SRqqt2/assignment_loss‚2.2895722488562265 0loss/FBt3/assignment_loss‚0.5788348255057656 /loss/FBt1/detection_loss‚2.5404854863882065 0loss/FBt1/assignment_loss‚0.3310555908817984 .loss/FRt4/detection_loss‚1.847963624068948 epoch‚2 _runtime‚2636.252466804 _step‚265 .loss/FRt2/detection_loss‚2.697318585589528 /loss/FRt3/detection_loss‚2.6592704340031275 0loss/SRqqt3/detection_loss‚1.727280578074547 1loss/SRqqt1/detection_loss‚2.4057363210580287 0loss/SRqqt2/detection_loss‚2.683167945455622 % _timestamp‚1.7883980116156762e+09 trainer/global_step‚3249 0loss/FRt3/assignment_loss‚2.7126759745572744 1loss/SRqqt4/assignment_loss‚1.214727429697584 1loss/FBt4/assignment_loss‚0.21963403374842477 /loss/FBt3/detection_loss‚2.6135265628496804ûê„^ðØ 1loss/SRqqt4/assignment_loss‚1.214727429697584 1loss/FBt4/assignment_loss‚0.21963403374842477 /loss/FBt3/detection_loss‚2.6135265628496804 %loss/total_loss‚48.43360711713672 /loss/FRt1/assignment_loss‚4.459208913569181 2loss/SRqqt1/assignment_loss‚2.0273797703094973 1loss/FBt2/assignment_loss‚0.34641309497806105 .loss/FRt1/detection_loss‚2.099075326942048 1loss/SRqqt4/detection_loss‚1.0292234779580642 2loss/SRqqt3/assignment_loss‚1.4262653368596847 /loss/FBt2/detection_loss‚3.7477521672844887 /loss/FRt2/assignment_loss‚4.305407457053661 0loss/FRt4/assignment_loss‚1.6875236239589628 /loss/FBt4/detection_loss‚0.7841083074883101 2loss/SRqqt2/assignment_loss‚2.2895722488562265 0loss/FBt3/assignment_loss‚0.5788348255057656 /loss/FBt1/detection_loss‚2.5404854863882065 0loss/FBt1/assignment_loss‚0.3310555908817984 .loss/FRt4/detection_loss‚1.847963624068948 epoch‚2 _runtime‚2636.252466804 _step‚265 .loss/FRt2/detection_loss‚2.697318585589528 /loss/FRt3/detection_loss‚2.6592704340031275 0loss/SRqqt3/detection_loss‚1.727280578074547 1loss/SRqqt1/detection_loss‚2.4057363210580287 0loss/SRqqt2/detection_loss‚2.683167945455622 % _timestamp‚1.7883980116156762e+09 trainer/global_step‚3249 0loss/FRt3/assignment_loss‚2.7126759745572744‰‚E ’ñ‚(:‡ ¿“ãÔ—ˆ¹¯ gpu.0.gpu‚42.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚111.883' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3766 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:uŽ¼jò‚(:ß ¿“ãÔ“‚± proc.cpu.threads‚17 network.sent‚ 7.059736e+06 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.sda.out‚ 6547.265625% disk./.usageGB‚249.56383514404297 disk.dm-0.out‚ 18765.5234375 disk.sda1.in‚0 disk.sda3.out‚ 6547.265625" disk.nvme0n1.out‚ 18765.5234375 disk.nvme0n1.in‚ 334.1484375 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 9.97265625( proc.memory.availableMB‚ 1398.1796875 disk.dm-1.in‚ 0.05078125 disk.dm-1.out‚ 6547.265625$ disk.nvme0c0n1.out‚ 18765.5234375 cpu‚25.304776885273075 disk.drbd2265.out‚0 disk.dm-0.in‚ 334.1484375 disk.drbd1716.in‚0$ memory_percent‚91.46618843078613# proc.memory.rssMB‚ 2693.83984375* proc.memory.percent‚16.441893577575684 network.recv‚962808* disk./.usagePercent‚16.748438374521736 disk.sda1.out‚0 disk.sda3.in‚ 0.05078125 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 334.1484375 disk.drbd2080.out‚0 disk.sda.in‚ 0.05078125 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0§|ñ­’ó‚(:‡ ΓãÔÀ® gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.527' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.7054 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e ùtô‚(:é ΓãÔä± disk.sda3.in‚ 0.05078125 disk.sda3.out‚ 6577.796875' proc.memory.availableMB‚ 1398.171875% disk./.usageGB‚249.56378173828125 disk.dm-1.in‚ 0.05078125 disk.dm-0.in‚ 334.1484375$ memory_percent‚91.46623611450195* proc.memory.percent‚16.441893577575684 cpu‚25.303351156305258 network.sent‚ 7.167452e+06 disk.sda2.in‚0 disk.drbd2266.in‚0& disk.nvme0n1.out‚18765.93603515625 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 6577.796875 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 6577.796875) disk./.usagePercent‚16.74843479041626 disk.drbd2080.in‚0 disk.sda1.in‚0! disk.nvme0c0n1.in‚ 334.1484375# proc.memory.rssMB‚ 2693.83984375 proc.cpu.threads‚17 network.recv‚968545 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚18765.93603515625 disk.nvme0n1.in‚ 334.1484375 disk.drbd2080.out‚0# disk.dm-0.out‚18765.93603515625 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 9.97265625 disk.sda2.out‚0 disk.sda.in‚ 0.05078125 disk.rbd0.in‚0Fn9Îõ‚(:’ Ý“ãÔ”µ® gpu.0.gpu‚27.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚135.089' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.017799999999998 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iAžIfö‚(:ò Ý“ãÔɢDz disk.sda1.in‚0 disk.drbd2266.in‚0 disk.drbd2265.in‚0 network.sent‚ 7.173437e+06 disk.rbd1.in‚0( disk.nvme0c0n1.out‚18766.15869140625 disk.drbd1716.out‚0 network.recv‚973065 disk.rbd1.out‚10.03564453125 disk.sda.in‚ 0.05078125 disk.sda2.out‚0) proc.memory.availableMB‚ 1398.16796875# proc.memory.rssMB‚ 2693.83984375 cpu‚25.302005504777384 disk.sda3.in‚ 0.05078125! disk.nvme0c0n1.in‚ 334.1484375 disk.dm-0.in‚ 334.1484375 disk.drbd2080.out‚0 disk.nvme0n1.in‚ 334.1484375 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.05078125 disk.dm-1.out‚ 6608.4765625 disk.drbd2265.out‚0 disk.rbd0.out‚0# disk.dm-0.out‚18766.15869140625 disk.sda3.out‚ 6608.4765625 disk.sda.out‚ 6608.4765625 disk.drbd2266.out‚0 disk.rbd0.in‚0$ memory_percˆâÇent‚91.46625995635986* proc.memory.percent‚16.441893577575684% disk./.usageGB‚249.56378173828125 disk.sda1.out‚0& disk.nvme0n1.out‚18766.15869140625 disk.sda2.in‚0 disk.drbd2080.in‚0) disk./.usagePercent‚16.74843479041626 proc.cpu.threads‚17 ™L2÷- _wandbruntime‚2672  _runtime‚2672›­8Éøà ' lr-AdamW/pg1‚0.00019600261678831416 ' lr-AdamW/pg2‚0.00019600261678831416 trainer/global_step‚3299 % _timestamp‚1.7883980477330475e+09 _runtime‚2672.369918065 _step‚266â½é¦ÎùÈ _runtime‚2672.369918065 _step‚266 ' lr-AdamW/pg1‚0.00019600261678831416 ' lr-AdamW/pg2‚0.00019600261678831416 trainer/global_step‚3299 % _timestamp‚1.7883980477330475e+09Š‰m~2ú- _wandbruntime‚2672  _runtime‚2672|õ%:Í'ûÇO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43133177 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97893967500288 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚87877974434816 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4628403249601 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109580550 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66467410 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66447373 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43133179 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25171306 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4628238128201 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚97020159870976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4641597628416¤€ÒÒ'üÌO E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43133179 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚92378580245504 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚92378562242560 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚102616736612864 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 % _timestamp‚1.7883980477337952e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚66447373 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚97020346873344 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚87877957395456 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚66467410 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚18650902 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚92506195523657 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚18650368 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚92506377684417 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚102616960652288 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚92378580245504 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4722769112576 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚66447373 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43133179 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚25171306 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4628238128201 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 Q>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚97020159870976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4641597628416 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚43133177 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚97893967500288 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚87877974434816 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4628403249601 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 109580550 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚66467410 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109633061 _step‚268 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43145838 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 trainer/global_step‚3299 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25178683 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚505 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚92533527870537 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43145836 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43145836 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚92405831607296 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66467160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚503 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4643159779328 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4629792414353 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7883980484271433e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚97049009389568 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚87903900577792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43834637 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43145838 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚102647298251264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚97049009389568 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4724185388032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66467160 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102647065836544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474624 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109612996 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4643159779328 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109612996 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4724393862656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4642990766592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18655956 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87903917617152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43835142 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚92405849610240 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97922904388608÷Yí¯ÿÕM L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7883980484271433e+09 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚97049009389568 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚87903900577792 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚43834637 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43145838 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚102647298251264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚97049009389568 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4724185388032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚66467160 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚102647065836544 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474624 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 109612996 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4643159779328 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 109612996 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4724393862656 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4642990766592 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18655956 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚87903917617152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚43835142 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚92405849610240 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚97922904388608 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚97922880448512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚97048822373888 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚92405831607296 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚66487223 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚66487223 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18656459 _runtime‚ 2673.06464787 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 109633061 _step‚268 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43145838 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 trainer/global_step‚3299 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25178683 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚505 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚92533527870537 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43145836 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43145836 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚92405831607296 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚66467160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚503 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4643159779328 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4629792414353 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965Œ ¦*H2€- _wandbruntime‚2673  _runtime‚2673½Y£ÓXÒ _runtime‚2673.065857013 /loss/FRt4/assignment_loss‚2.187666074434916 % _timestamp‚1.7883980484290946e+09 .loss/FRt1/detection_loss‚2.100205522996408 0loss/FRt1/assignment_loss‚3.7803026261153043 1loss/SRqqt2/assignment_loss‚2.470549061195925 0loss/FBt4/assignment_loss‚0.3071873080063294 /loss/FRt3/detection_loss‚2.8389114018748787 /loss/FRt4/detection_loss‚1.8027697498599689 _step‚269 /loss/SRqqt1/detection_loss‚1.70154719799757 /loss/FRt3/assignment_loss‚3.101360920597525 2loss/SRqqt1/assignment_loss‚1.8924826722192978 /loss/FRt2/assignment_loss‚4.362118695721482 0loss/FBt1/assignment_loss‚0.5308529419193152 1loss/FBt2/assignment_loss‚0.38505046002359855 /loss/FBt1/detection_loss‚2.5769171164585996 2loss/SRqqt3/assignment_loss‚1.8221130762614457 2loss/SRqqt4/assignment_loss‚1.2711482628395683 0loss/FBt3/assignment_loss‚0.4295152988698747 1loss/SRqqt2/detection_loss‚2.2547737266868353 trainer/global_step‚3299 /loss/FRt2/detection_loss‚2.6911415728655728 0loss/SRqqt3/detection_loss‚1.816261208524891 /loss/FBt2/detection_loss‚3.3224250144428678 /loss/FBt3/detection_loss‚2.8164500726593866 1loss/SRqqt4/detection_loss‚1.1354519982675189 /loss/FBt4/detection_loss‚0.7543851071466633 %loss/total_loss‚48.35158708798574 epoch‚2æ;L]‚× 2loss/SRqqt3/assignment_loss‚1.8221130762614457 2loss/SRqqt4/assignment_loss‚1.2711482628395683 0loss/FBt3/assignment_loss‚0.4295152988698747 1loss/SRqqt2/detection_loss‚2.2547737266868353 trainer/global_step‚3299 /loss/FRt2/detection_loss‚2.6911415728655728 0loss/SRqqt3/detection_loss‚1.816261208524891 /loss/FBt2/detection_loss‚3.3224250144428678 /loss/FBt3/detection_loss‚2.8164500726593866 1loss/SRqqt4/detection_loss‚1.1354519982675189 /loss/FBt4/detection_loss‚0.7543851071466633 %loss/total_loss‚48.35158708798574 epoch‚2 _runtime‚2673.065857013 /loss/FRt4/assignment_loss‚2.187666074434916 % _timestamp‚1.7883980484290946e+09 .loss/FRt1/detection_loss‚2.100205522996408 0loss/FRt1/assignment_loss‚3.7803026261153043 1loss/SRqqt2/assignment_loss‚2.470549061195925 0loss/FBt4/assignment_loss‚0.3071873080063294 /loss/FRt3/detection_loss‚2.8389114018748787 /loss/FRt4/detection_loss‚1.8027697498599689 _step‚269 /loss/SRqqt1/detection_loss‚1.70154719799757 /loss/FRt3/assignment_loss‚3.101360920597525 2loss/SRqqt1/assignment_loss‚1.8924826722192978 /loss/FRt2/assignment_loss‚4.362118695721482 0loss/FBt1/assignment_loss‚0.5308529419193152 1loss/FBt2/assignment_loss‚0.38505046002359855 /loss/FBt1/detection_loss‚2.5769171164585996h`Ⴢ(:’ ì“ãÔ°® gpu.0.gpu‚30.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚141.047' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.209400000000002 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žŸ€Ÿt„‚(:é ì“ãÔÏò±°) proc.memory.availableMB‚ 1398.09765625# proc.memory.rssMB‚ 2693.83984375 disk.rbd0.in‚0 network.recv‚977302% disk./.usageGB‚249.56361389160156 disk.drbd1716.out‚0 disk.dm-1.out‚ 6640.859375 disk.drbd2080.out‚0 disk.rbd0.out‚0! disk.dm-0.out‚18766.482421875 disk.rbd1.in‚0& disk.nvme0c0n1.out‚18766.482421875 disk.drbd1716.in‚0 disk.sda.in‚ 0.05078125 disk.sda2.in‚0 network.sent‚ 7.179429e+06 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.05078125$ disk.nvme0n1.out‚18766.482421875 cpu‚25.30056050703981* disk./.usagePercent‚16.748423526084753 disk.drbd2265.out‚0 disk.dm-0.in‚ 334.1484375 disk.sda2.out‚0$ memory_percent‚91.46668910980225* proc.memory.percent‚16.441893577575684 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.sda3.out‚ 6640.859375 disk.rbd1.out‚10.03564453125! disk.nvme0c0n1.in‚ 334.1484375 disk.dm-1.in‚ 0.05078125 disk.drbd2266.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 334.1484375 disk.sda.out‚ 6640.859375_•·ñ’…‚(:‡ û“ãÔû°Š® gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚135.619' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.1238 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ûð%Vv†‚(:ë û“ãÔý¯ì° cpu‚25.2991723718662! disk.nvme0c0n1.in‚ 334.1484375( disk.nvme0c0n1.out‚18767.00927734375& disk.nvme0n1.out‚18767.00927734375 proc.cpu.threads‚17 disk.sda.out‚ 6667.609375 network.sent‚ 7.288164e+06* disk./.usagePercent‚16.748427366197767 disk.dm-0.in‚ 334.1484375 disk.rbd1.in‚0 disk.sda.in‚ 0.05078125 network.recv‚984007% disk./.usageGB‚249.56367111206055 disk.dm-1.out‚ 6667.609375 disk.drbd2265.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.rbd1.out‚10.03564453125 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0* proc.memory.percent‚16.441893577575684 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.nvme0n1.in‚ 334.1484375$ memory_percent‚91.46709442138672# proc.memory.rssMB‚ 2693.83984375 disk.drbd2080.out‚0 disk.rbd0.in‚0# disk.dm-0.out‚18767.00927734375 disk.sda3.in‚ 0.05078125& proc.memory.availableMB‚ 1398.03125 disk.dm-1.in‚ 0.05078125 disk.sda1.out‚0 disk.sda3.out‚ 6667.609375¾0ÌÕ2‡- _wandbruntime‚2709  _runtime‚2709¬ÉˆÃ ' lr-AdamW/pg2‚0.00019511283055542944 trainer/global_step‚3349 % _timestamp‚1.7883980846927185e+09 _runtime‚2709.329604055 _step‚270 ' lr-AdamW/pg1‚0.000195112830555429449 œÎ‰È % _timestamp‚1.7883980846927185e+09 _runtime‚2709.329604055 _step‚270 ' lr-AdamW/pg1‚0.00019511283055542944 ' lr-AdamW/pg2‚0.00019511283055542944 trainer/global_step‚3349Ž<Ä€«2Š- _wandbruntime‚2709  _runtime‚2709ÌÍÍ'‹ÇO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚104133479531520 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚43766129 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4711268951040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4793636386304 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43766129 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4711437950464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7883980846934764e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚98453486432768 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚93873008619625 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚99339614911488 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚93742048482304 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111226241 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚104133251297792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚93872826458865 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚93742030479360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98453299430400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44462942 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚93742048482304 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 111226241 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚104133251297792 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚93872826458865 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚93742030479360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚98453299430400 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚44462942 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4711268951040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4793636386304 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚43766129 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4711437950464 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 % _timestamp‚1.7883980846934764e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037HâK¶2- _wandbruntime‚2710  _runtime‚2710ø”ÃÀ×&ŽÑM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚93769317847040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98482148948992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚506 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚104163811864064 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414720 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4712831101952 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚89201059694592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44475708 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚508 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18927669 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4699264232761 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67459862 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18928175 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 _runtime‚2710.025614638 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98481961933312 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚93769299844096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4795051589632 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4712662089216 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4712662089216 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111258713 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43778788 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚104163579449344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43778788 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44475200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4795260064256 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚93900340966713 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚93900158805745 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43778786 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 % _timestamp‚1.7883980853881464e+09 _step‚272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67479925 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚99368551799808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98482148948992 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚99368527859712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111238648 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4699099111153 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚67479925 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 111258713 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚93769317847040 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25547533 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚89201076733952 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43778786 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67459862 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚89201076733952 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚43778786 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚67459862 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚93769317847040 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 S@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚98482148948992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚506 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚104163811864064 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414720 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4712831101952 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚89201059694592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚44475708 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚508 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚18927669 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4699264232761 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚67459862 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚18928175 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 _runtime‚2710.025614638 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚98481961933312 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚93769299844096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4795051589632 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4712662089216 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4712662089216 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 111258713 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚43778788 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚104163579449344 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚43778788 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚44475200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4795260064256 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚93900340966713 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚93900158805745 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚43778786 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 % _timestamp‚1.7883980853881464e+09 _step‚272 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚67479925 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚99368551799808 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚98482148948992 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚99368527859712 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 111238648 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4699099111153·Ê¸P2- _wandbruntime‚2710  _runtime‚2710i^Lä\‘Ö 1loss/SRqqt3/detection_loss‚1.8806737998257512 /loss/FRt4/detection_loss‚1.8643513616513123 % _timestamp‚1.7883980853900163e+09 /loss/FRt3/assignment_loss‚3.295945320059271 2loss/SRqqt1/assignment_loss‚1.8142953356558627 2loss/SRqqt4/assignment_loss‚1.1293778276333633 1loss/FBt2/assignment_loss‚0.27001598849892616 0loss/FRt4/assignment_loss‚1.3610553714362057 /loss/FRt1/detection_loss‚1.9619547224813891 /loss/FBt2/detection_loss‚3.3526332527399063 1loss/FBt1/assignment_loss‚0.10096705140458653 _step‚273 .loss/FBt1/detection_loss‚4.316691839694977 epoch‚2 trainer/global_step‚3349 _runtime‚2710.026776542 .loss/FRt3/detection_loss‚2.576146727975677 1loss/SRqqt1/detection_loss‚1.6484985251318325 1loss/SRqqt2/detection_loss‚2.5531567217725697 1loss/SRqqt4/detection_loss‚1.1088854757448037 0loss/FBt3/assignment_loss‚0.1361440937810888 0loss/FRt1/assignment_loss‚3.9143727440987863 /loss/FRt2/assignment_loss‚4.688192901611328 1loss/SRqqt2/assignment_loss‚2.049116071200732 /loss/FBt4/detection_loss‚0.8382282609961944 %loss/total_loss‚49.52709825930945 1loss/FBt4/assignment_loss‚0.18927639615743763 .loss/FRt2/detection_loss‚3.137447352409363 /loss/FBt3/detection_loss‚3.8397387663523355 2loss/SRqqt3/assignment_loss‚1.4999323509957478†ªÕa’Û /loss/FRt4/detection_loss‚1.8643513616513123 % _timestamp‚1.7883980853900163e+09 /loss/FRt3/assignment_loss‚3.295945320059271 2loss/SRqqt1/assignment_loss‚1.8142953356558627 2loss/SRqqt4/assignment_loss‚1.1293778276333633 1loss/FBt2/assignment_loss‚0.27001598849892616 0loss/FRt4/assignment_loss‚1.3610553714362057 /loss/FRt1/detection_loss‚1.9619547224813891 /loss/FBt2/detection_loss‚3.3526332527399063 1loss/FBt1/assignment_loss‚0.10096705140458653 _step‚273 .loss/FBt1/detection_loss‚4.316691839694977 epoch‚2 trainer/global_step‚3349 _runtime‚2710.026776542 .loss/FRt3/detection_loss‚2.576146727975677 1loss/SRqqt1/detection_loss‚1.6484985251318325 1loss/SRqqt2/detection_loss‚2.5531567217725697 1loss/SRqqt4/detection_loss‚1.1088854757448037 0loss/FBt3/assignment_loss‚0.1361440937810888 0loss/FRt1/assignment_loss‚3.9143727440987863 /loss/FRt2/assignment_loss‚4.688192901611328 1loss/SRqqt2/assignment_loss‚2.049116071200732 /loss/FBt4/detection_loss‚0.8382282609961944 %loss/total_loss‚49.52709825930945 1loss/FBt4/assignment_loss‚0.18927639615743763 .loss/FRt2/detection_loss‚3.137447352409363 /loss/FBt3/detection_loss‚3.8397387663523355 2loss/SRqqt3/assignment_loss‚1.4999323509957478 1loss/SRqqt3/detection_loss‚1.8806737998257512‘ý¯’“‚(:‡ Š”ãÔ¼ñ•® gpu.0.gpu‚21.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚103.021' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.6042 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 _/{”‚(:ð Š”ãÔ½…˜± disk.nvme0n1.in‚ 334.1484375 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 334.1484375 disk.sda3.in‚ 0.05078125 disk.rbd1.in‚0 disk.sda.in‚ 0.05078125 proc.cpu.threads‚17 network.sent‚ 7.293514e+06% disk./.usageGB‚249.56407165527344 disk.drbd2265.in‚0 disk.drbd1716.out‚0( disk.nvme0c0n1.out‚18767.72900390625 disk.drbd1716.in‚0' proc.memory.availableMB‚ 1397.984375* proc.memory.percent‚16.441893577575684 disk.rbd0.out‚0 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.sda2.in‚0 cpu‚25.297802155386215 network.recv‚987717 disk.dm-1.out‚ 6693.1171875 disk.drbd2266.out‚0! disk.nvme0c0n1.in‚ 334.1484375& disk.nvme0n1.out‚18767.72900390625 disk.sda.out‚ 6693.1171875# disk.dm-0.out‚18767.72900390625 disk.sda3.out‚ 6693.1171875 disk.rbd1.out‚10.03564453125$ memory_percent‚91.46738052368164 disk.sda1.out‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2693.83984375) disk./.usagePercent‚16.74845424698886 disk.dm-1.in‚ 0.05078125 disk.rbd0.in‚0 disk.sda1.in‚0{’£i•‚(:’ ™”ãÔÓî™® gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚135.908' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.181599999999996 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0÷ä>ßm–‚(:â ™”ãÔ×Ô³±( disk.nvme0c0n1.out‚18767.94775390625 disk.sda.in‚ 0.05078125 disk.drbd2266.out‚0# disk.dm-0.out‚18767.94775390625 disk.sda1.in‚0 disk.sda2.in‚0$ memory_percent‚91.47038459777832 network.recv‚993929 disk.drbd2080.out‚0 disk.sda3.in‚ 0.05078125# proc.memory.rssMB‚ 2693.83984375 network.sent‚ 7.401905e+06 disk.drbd2080.in‚0 disk.sda3.out‚6719.75 disk.rbd1.in‚0 disk.rbd1.out‚10.12158203125 disk.nvme0n1.in‚ 334.1484375 disk.drbd2266.in‚0 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.05078125* disk./.usagePercent‚16.748458343109405$ disk./.usageGB‚249.5641326904297 disk.sda1.out‚0& disk.nvme0n1.out‚18767.94775390625 disk.drbd2265.in‚0( proc.memory.availableMB‚ 1397.4921875 disk.rbd0.in‚0 disk.sda.out‚6719.75 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.sda2.out‚0 disk.drbd2265.out‚0 cpu‚25.296448394558748 disk.drbd1716.out‚0* proc.memory.percent‚16.441893577575684 disk.dm-1.out‚6719.75 disk.dm-0.in‚ 334.1484375! disk.nvme0c0n1.in‚ 334.1484375”œP—‚(:’ ¨”ãÔÙ¢® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚142.106' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.421199999999995 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è‰ùy˜‚(:î ¨”ãÔó×´° disk.drbd2265.out‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚18768.2900390625 disk.sda1.out‚0 disk.sda.out‚ 6747.6796875) proc.memory.availableMB‚ 1397.93359375# proc.memory.rssMB‚ 2693.83984375 disk.sda1.in‚0 disk.rbd1.out‚10.20751953125 disk.rbd0.in‚0" disk.dm-0.out‚18768.2900390625 disk.sda3.out‚ 6747.6796875! disk.nvme0c0n1.in‚ 334.1484375% disk.nvme0n1.out‚18768.2900390625 disk.sda2.out‚0* proc.memory.percent‚16.441893577575684 network.recv‚998144 disk.drbd1716.out‚0 disk.drbd2265.in‚0 cpu‚25.29513267996996 network.sent‚ 7.407796e+06* disk./.usagePercent‚16.748458343109405 disk.nvme0n1.in‚ 334.1484375 disk.dm-1.out‚ 6747.6796875 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0$ disk./.usageGB‚249.5641326904297 disk.dm-0.in‚ 334.1484375 disk.sda3.in‚ 0.05078125 disk.sda2.in‚0 disk.dm-1.in‚ 0.05078125$ memory_percent‚91.46769046783447 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.sda.in‚ 0.05078125:Ãã2™- _wandbruntime‚2746  _runtime‚2746^®o1ɚà _step‚274 ' lr-AdamW/pg1‚0.00019420478454512555 ' lr-AdamW/pg2‚0.00019420478454512555 trainer/global_step‚3399 % _timestamp‚1.7883981216731062e+09 _runtime‚2746.309945707s!üËÎ›È _step‚274 ' lr-AdamW/pg1‚0.00019420478454512555 ' lr-AdamW/pg2‚0.00019420478454512555 trainer/global_step‚3399 % _timestamp‚1.7883981216731062e+09 _runtime‚2746.309945707’¦Z¸<2œ- _wandbruntime‚2746  _runtime‚2746CÒLΰÉO JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45103528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19194524 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚99886625992192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45104049 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4767346886417 _runtime‚2746.310912452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚99886438989824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44399077 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19195043 cQDeviceStatsMonitor.on_trai‚NÐän_batch_start/inactive_split_bytes.small_pool.allocated‚ 4864725588480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚519 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚99886625992192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4781109273088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚95239457394073 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90472292668416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 trainer/global_step‚3399 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105649777326080 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68452814 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68432777 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99886438989824 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4780940273664 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚100785262322688 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4780940273664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚100785286262784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4864515003392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4781109273088 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210585088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4767181765017 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 112831854 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44399079 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44399077 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚95105516719104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44399079 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚95105516719104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 112851893 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105650011851264 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90472275629056 % _timestamp‚1.7883981216738493e+09 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚45103528 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19194524 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 XEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚99886625992192 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45104049 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4767346886417 _runtime‚2746.310912452 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚99886438989824 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚44399077 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19195043 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4864725588480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚519 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚99886625992192 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4781109273088 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚95239457394073 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚90472292668416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 trainer/global_step‚3399 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚105649777326080 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚68452814 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚68432777 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚99886438989824 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4780940273664 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚100785262322688 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4780940273664 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚100785286262784 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 4864515003392 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4781109273088 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210585088 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4767181765017 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 112831854 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚44399079 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚44399077 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚95105516719104 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚44399079 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚95105516719104 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 112851893 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚105650011851264 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚90472275629056 % _timestamp‚1.7883981216738493e+09 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 221057536 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45116241 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44411736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19199860 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25916381 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4782333411840 _runtime‚ 2747.00997832 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 _step‚276 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚95132786083840 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4782502424576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4866148260864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚95266789740953 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68472627 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7883981223726485e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 112884365 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112884365 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68452564 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚95132786083840 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105680102474240 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚95132768080896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚100814199211008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19200386 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚95132768080896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68472627 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25916383 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44411736 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚99915288508416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4865927203328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4782502424576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚99915101492736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112864300 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44411738 O6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚90498218811392 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26ÚšvSà¡×M \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4768736051169 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚90498218811392 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45116769 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚68452564 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 221057536 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45116241 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚44411736 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19199860 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚25916381 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4782333411840 _runtime‚ 2747.00997832 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 _step‚276 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚95132786083840 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4782502424576 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4866148260864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 R?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚95266789740953 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚68472627 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7883981223726485e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 112884365 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 112884365 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚68452564 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚95132786083840 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚105680102474240 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚95132768080896 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚100814199211008 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚19200386 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚95132768080896 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚68472627 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚25916383 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚44411736 VCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚99915288508416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4865927203328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4782502424576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 R?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚99915101492736 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 112864300 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚44411738 ODeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚101319578549248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622336 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114457506 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4836653583425 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚101319765551616 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4850611596288 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45032027 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4836818704825 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96606088329281 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚96468966952960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26277856 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45032029 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69445516 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4935592211968 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69445516 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚563 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114477545 _step‚279 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚91769434745856 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45032029 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚96468966952960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19465104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4850780595712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19465665 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚102230909733888 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚96606270490041 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚96468984955904 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚102230933673984 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚96468984955904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚107166289263616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚107166525885952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚101319578549248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 % _timestamp‚1.7883981589265122e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚101319765551616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199¤ÎŠØ'®ÒO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚101319765551616 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4850611596288 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚45032027 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4836818704825 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚96606088329281 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚96468966952960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212682240 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26277856 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚45032029 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚69445516 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 4935592211968 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚69445516 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚563 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 114477545 _step‚279 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚91769434745856 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45032029 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚96468966952960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19465104 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4850780595712 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚19465665 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚102230909733888 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚96606270490041 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚96468984955904 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚102230933673984 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚96468984955904 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚107166289263616 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚107166525885952 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚101319578549248 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 % _timestamp‚1.7883981589265122e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚101319765551616 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _runtime‚2783.563588811 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚45743521 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚561 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚69425479 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚91769451785216 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚101319578549248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236622336 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 114457506 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4836653583425—uî»2¯- _wandbruntime‚2784  _runtime‚2784B÷õ­°×M trainer/global_step‚3449 % _timestamp‚1.7883981596252024e+09 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚101348241052160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 4.DeviceStatsMonito&àpÀ0%r.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69445266 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102259846622208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4936797384192 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69465329 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚96496254320640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69465329 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚96496236317696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚107196856675328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69445266 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚102259822682112 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45756165 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4838207869577 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114489952 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚91795394967552 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4852173747200 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4937010053120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45044688 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45044686 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚96496254320640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4852004734464 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45755651 _step‚280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4852173747200 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚101348428067840 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114510017 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248Fγâ&±ÜM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚96496254320640 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4852004734464 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚45755651 _step‚280 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4852173747200 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚101348428067840 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 114510017 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 trainer/global_step‚3449 % _timestamp‚1.7883981596252024e+09 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚101348241052160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚69445266 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚102259846622208 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 4936797384192 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚69465329 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚96496254320640 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚69465329 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚96496236317696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚107196856675328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚69445266 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚102259822682112 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚45756165 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4838207869577 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 114489952 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚91795394967552 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4852173747200 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 4937010053120 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚45044688 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚45044686 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480˜Š2]2²- _wandbruntime‚2784  _runtime‚2784åÆ!X³Ò /loss/FRt1/detection_loss‚2.0748113482086747 %loss/total_loss‚47.34337390497214 1loss/SRqqt2/assignment_loss‚2.386381631717086 0loss/FBt3/assignment_loss‚0.5298251713703311 -loss/FBt2/detection_loss‚2.14654303714633 2loss/SRqqt4/assignment_loss‚1.2944572514284458 /loss/FRt4/detection_loss‚1.6225278932721383 epoch‚2 /loss/FRt2/assignment_loss‚4.016108180085818 1loss/SRqqt1/detection_loss‚1.9129326301951741 .loss/FBt1/detection_loss‚3.465980604290962 trainer/global_step‚3449 0loss/FRt1/assignment_loss‚3.6000346221305706 2loss/SRqqt1/assignment_loss‚1.6827725917100906 2loss/SRqqt3/assignment_loss‚1.4289960690053567 /loss/FRt3/detection_loss‚2.3206886996825538 _step‚281 /loss/FRt4/assignment_loss‚2.075351530505765 1loss/FBt4/assignment_loss‚0.20655055388268034 1loss/SRqqt2/detection_loss‚2.9979933376113572 /loss/FBt4/detection_loss‚0.6982763036008105 _runtime‚2784.264171212 .loss/FRt2/detection_loss‚3.243035927414894 1loss/SRqqt3/detection_loss‚1.8720729722166962 .loss/FBt3/detection_loss‚2.821285394105044 % _timestamp‚1.7883981596274247e+09 0loss/FRt3/assignment_loss‚2.5548947964395796 1loss/SRqqt4/detection_loss‚1.2592433747182141 0loss/FBt1/assignment_loss‚0.5760228652507067 0loss/FBt2/assignment_loss‚0.5565871189828613©¥.)]´× 1loss/SRqqt2/assignment_loss‚2.386381631717086 0loss/FBt3/assignment_loss‚0.5298251713703311 -loss/FBt2/detection_loss‚2.14654303714633 2loss/SRqqt4/assignment_loss‚1.2944572514284458 /loss/FRt4/detection_loss‚1.6225278932721383 epoch‚2 /loss/FRt2/assignment_loss‚4.016108180085818 1loss/SRqqt1/detection_loss‚1.9129326301951741 .loss/FBt1/detection_loss‚3.465980604290962 trainer/global_step‚3449 0loss/FRt1/assignment_loss‚3.6000346221305706 2loss/SRqqt1/assignment_loss‚1.6827725917100906 2loss/SRqqt3/assignment_loss‚1.4289960690053567 /loss/FRt3/detection_loss‚2.3206886996825538 _step‚281 /loss/FRt4/assignment_loss‚2.075351530505765 1loss/FBt4/assignment_loss‚0.20655055388268034 1loss/SRqqt2/detection_loss‚2.9979933376113572 /loss/FBt4/detection_loss‚0.6982763036008105 _runtime‚2784.264171212 .loss/FRt2/detection_loss‚3.243035927414894 1loss/SRqqt3/detection_loss‚1.8720729722166962 .loss/FBt3/detection_loss‚2.821285394105044 % _timestamp‚1.7883981596274247e+09 0loss/FRt3/assignment_loss‚2.5548947964395796 1loss/SRqqt4/detection_loss‚1.2592433747182141 0loss/FBt1/assignment_loss‚0.5760228652507067 0loss/FBt2/assignment_loss‚0.5565871189828613 /loss/FRt1/detection_loss‚2.0748113482086747 %loss/total_loss‚47.34337390497214™¬*€zµ‚(:’ Õ”ãÔ†þŸ® gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚127.824' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.564799999999998 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±P»‚¶‚(:÷ Õ”ãԉ㱠disk.sda.out‚ 6821.48046875 disk.drbd1716.out‚0) disk./.usagePercent‚16.74845859911694 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.dm-0.in‚ 334.1484375 disk.sda3.out‚ 6821.48046875% disk.nvme0n1.out‚18769.0849609375 disk.drbd2266.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.05078125 disk.dm-1.out‚ 6821.48046875 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 334.1484375( proc.memory.availableMB‚ 1397.8203125 network.recv‚ 1.011875e+06 disk.sda3.in‚ 0.05078125! disk.nvme0c0n1.in‚ 334.1484375' disk.nvme0c0n1.out‚18769.0849609375 proc.cpu.threads‚17 network.sent‚ 7.527623e+06 disk.sda.in‚ 0.05078125 disk.sda2.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0* proc.memory.percent‚16.441893577575684% disk./.usageGB‚249.56413650512695 disk.drbd2265.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2693.83984375 cpu‚25.291202646251982 disk.rbd0.out‚0 disk.rbd1.in‚0" disk.dm-0.out‚18769.0849609375 disk.rbd1.out‚10.30224609375$ memory_percent‚91.46838188171387FèŠ{’·‚(:‡ ä”ãÔç“©® gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚117.552' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.5104 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0YÑÓ€~¸‚(:ó ä”ãÔŠ¹º° disk.dm-1.out‚ 6847.5859375 disk.drbd2080.in‚0 disk.rbd0.in‚0$ memory_percent‚91.46838188171387 disk.rbd1.out‚10.30224609375( proc.memory.availableMB‚ 1397.8203125* proc.memory.percent‚16.441893577575684 cpu‚25.28993458657183 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2693.83984375 proc.cpu.threads‚17% disk./.usageGB‚249.56413650512695 disk.sda3.in‚ 0.05078125! disk.nvme0c0n1.in‚ 334.1484375 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd2266.out‚0 network.sent‚ 7.636069e+06 disk.sda.in‚ 0.05078125 disk.drbd2265.in‚0 disk.drbd2080.out‚0" disk.dm-0.out‚18769.2333984375 disk.sda1.out‚0) disk./.usagePercent‚16.74845859911694' disk.nvme0c0n1.out‚18769.2333984375% disk.nvme0n1.out‚18769.2333984375 disk.drbd2265.out‚0 network.recv‚ 1.018192e+06 disk.sda3.out‚ 6847.5859375 disk.rbd1.in‚0 disk.dm-1.in‚ 0.05078125 disk.dm-0.in‚ 334.1484375 disk.nvme0n1.in‚ 334.1484375 disk.sda.out‚ 6847.5859375ô˜¦A¹‚(:’ ó”ãÔ¨¤ª® gpu.0.gpu‚22.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚125.402' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.080400000000004 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µ`þ‡~º‚(:ó ó”ãÔ…즱 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 6874.05078125! disk.nvme0c0n1.in‚ 334.1484375 disk.sda.out‚ 6874.05078125 disk.dm-1.out‚ 6874.05078125 disk.rbd0.out‚0' disk.nvme0c0n1.out‚18769.5771484375 disk.sda2.in‚0" disk.dm-0.out‚18769.5771484375 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 10.365234375 disk.drbd1716.out‚0 cpu‚25.28865409790115 disk.dm-0.in‚ 334.1484375% disk.nvme0n1.out‚18769.5771484375 disk.drbd1716.in‚0* proc.memory.percent‚16.441893577575684 disk.rbd0.in‚0% disk./.usageGB‚249.56419372558594 disk.drbd2266.in‚0 disk.sda3.in‚ 0.05078125 disk.sda.in‚ 0.05078125* disk./.usagePercent‚16.748462439229954 disk.sda1.in‚0$ memory_percent‚91.46862030029297# proc.memory.rssMB‚ 2693.83984375 proc.cpu.threads‚17 disk.nvme0n1.in‚ 334.1484375 disk.sda2.out‚0 network.sent‚ 7.642369e+06 network.recv‚ 1.023255e+06 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.dm-1.in‚ 0.05078125& proc.memory.availableMB‚ 1397.78125;à=Ö2»- _wandbruntime‚2820  _runtime‚2820Îôóɼà ' lr-AdamW/pg1‚0.00019233467389984416 ' lr-AdamW/pg2‚0.00019233467389984416 trainer/global_step‚3499 % _timestamp‚1.7883981959461129e+09 _runtime‚2820.582972134 _step‚282ÇÝ ‰Î½È % _timestamp‚1.7883981959461129e+09 _runtime‚2820.582972134 _step‚282 ' lr-AdamW/pg1‚0.00019233467389984416 ' lr-AdamW/pg2‚0.00019233467389984416 trainer/global_step‚3499š§y– 2¾- _wandbruntime‚2820  _runtime‚2820…*R¿ÌO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108682814532096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4906125401833 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4920451918336 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5006257387008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108683046960128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚97832435189760 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚102752718108672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚97972719264489 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4906290523233 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116083158 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70438218 $ _timestamp‚1.788398195946885e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45664977 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚97832435189760 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚93066593862656 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45664979 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚102752718108672 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚97832453192704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚102752905111040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26646704 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70438218 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19738983 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4920282918912 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26646706 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232428032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚2820.583958095 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚97832453192704 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116103197 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480ÀÅhS×'ÀÑO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚102752718108672 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚97972719264489 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4906290523233 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 116083158 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚70438218 $ _timestamp‚1.788398195946885e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚45664977 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚97832435189760 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚93066593862656 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚45664979 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚102752718108672 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚97832453192704 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚102752905111040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20039 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚26646704 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚70438218 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚19738983 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4920282918912 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚26646706 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232428032 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚2820.583958095 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚97832453192704 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 116103197 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚108682814532096 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4906125401833 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4920451918336 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5006257387008 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚108683046960128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚97832435189760›cǦÝ2Á- _wandbruntime‚2821  _runtime‚2821s³M£Ü&ÂÖM REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4921676057088 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46398425 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45677636 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474624 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚485 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚483 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4921676057088 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚93092537044992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚103705494033408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 $ _timestamp‚1.788398196640216e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116115604 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45677638 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚97859704554496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚103705470093312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26654081 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚284 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4907514566377 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _runtime‚2821.277650268 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚97859704554496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 trainer/global_step‚3499 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚97859722557440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4921845069824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26654083 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚102781567627264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚102781567627264 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116135669 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19744344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5007676121088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4907679687985˶m¬xÃÛM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚45677636 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474624 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚485 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚483 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4921676057088 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚93092537044992 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414720 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚103705494033408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 $ _timestamp‚1.788398196640216e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 116115604 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚45677638 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚97859704554496 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚103705470093312 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚26654081 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚284 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4907514566377 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.pe 00i ak‚217 _runtime‚2821.277650268 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚97859704554496 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 trainer/global_step‚3499 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚97859722557440 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 4921845069824 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚26654083 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚102781567627264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚102781567627264 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 116135669 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚19744344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5007676121088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4907679687985 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4921676057088 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚46398425 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0œÿ^ 2Ä- _wandbruntime‚2821  _runtime‚2821 43XÅÒ 0loss/FRt1/assignment_loss‚3.6423302663947053 trainer/global_step‚3499 1loss/FBt2/assignment_loss‚0.43895515346860825 .loss/FRt2/detection_loss‚3.220885344173597 epoch‚2 /loss/FRt2/assignment_loss‚5.219991673593936 0loss/FBt3/assignment_loss‚0.5735638705746494 /loss/FRt1/detection_loss‚2.1509994931940764 /loss/FRt3/detection_loss‚2.8765676861459557 1loss/SRqqt4/detection_loss‚1.1301896701256433 .loss/FBt2/detection_loss‚2.601891303062439 /loss/FBt4/detection_loss‚0.7011618378195573 0loss/FBt4/assignment_loss‚0.2489143697764276 1loss/SRqqt1/detection_loss‚1.8314318178959612 /loss/FBt3/detection_loss‚2.7800328646387373 0loss/FRt3/assignment_loss‚3.7797318263487383 % _timestamp‚1.7883981966421518e+09 _runtime‚2821.278935544 0loss/FRt4/assignment_loss‚1.6289039266352752 2loss/SRqqt3/assignment_loss‚1.5240502018554538 /loss/FRt4/detection_loss‚1.8494203768822612 1loss/SRqqt3/detection_loss‚1.6842381065967036 %loss/total_loss‚48.08402488544915 1loss/SRqqt1/assignment_loss‚1.547491335187037 1loss/SRqqt4/assignment_loss‚1.201249897579352 /loss/FBt1/assignment_loss‚0.487907981138657 0loss/SRqqt2/detection_loss‚3.009518912544957 /loss/FBt1/detection_loss‚2.0859504813949266 _step‚285 1loss/SRqqt2/assignment_loss‚1.868646488421493üMv]Æ× 0loss/FRt3/assignment_loss‚3.7797318263487383 % _timestamp‚1.7883981966421518e+09 _runtime‚2821.278935544 0loss/FRt4/assignment_loss‚1.6289039266352752 2loss/SRqqt3/assignment_loss‚1.5240502018554538 /loss/FRt4/detection_loss‚1.8494203768822612 1loss/SRqqt3/detection_loss‚1.6842381065967036 %loss/total_loss‚48.08402488544915 1loss/SRqqt1/assignment_loss‚1.547491335187037 1loss/SRqqt4/assignment_loss‚1.201249897579352 /loss/FBt1/assignment_loss‚0.487907981138657 0loss/SRqqt2/detection_loss‚3.009518912544957 /loss/FBt1/detection_loss‚2.0859504813949266 _step‚285 1loss/SRqqt2/assignment_loss‚1.868646488421493 0loss/FRt1/assignment_loss‚3.6423302663947053 trainer/global_step‚3499 1loss/FBt2/assignment_loss‚0.43895515346860825 .loss/FRt2/detection_loss‚3.220885344173597 epoch‚2 /loss/FRt2/assignment_loss‚5.219991673593936 0loss/FBt3/assignment_loss‚0.5735638705746494 /loss/FRt1/detection_loss‚2.1509994931940764 /loss/FRt3/detection_loss‚2.8765676861459557 1loss/SRqqt4/detection_loss‚1.1301896701256433 .loss/FBt2/detection_loss‚2.601891303062439 /loss/FBt4/detection_loss‚0.7011618378195573 0loss/FBt4/assignment_loss‚0.2489143697764276 1loss/SRqqt1/detection_loss‚1.8314318178959612 /loss/FBt3/detection_loss‚2.7800328646387373âÆÇ‚(:’ ‚•ãÔ›°´® gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚134.552' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.910400000000003 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0íì€È‚(:õ ‚•ãÔÚ©± disk.dm-0.in‚ 334.1484375" disk.dm-0.out‚18769.8505859375 disk.sda3.in‚ 0.05078125 disk.rbd1.out‚ 10.365234375! disk.nvme0c0n1.in‚ 334.1484375 disk.drbd2265.in‚0 network.sent‚ 7.647819e+06 disk.sda1.out‚0 disk.rbd1.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0* proc.memory.percent‚16.441893577575684 proc.cpu.threads‚17 network.recv‚ 1.026686e+06 disk.drbd1716.in‚0% disk./.usageGB‚249.56413650512695 disk.rbd0.out‚0 disk.dm-1.out‚ 6902.29296875 disk.drbd2080.in‚0# proc.memory.rssMB‚ 2693.83984375 cpu‚25.287314465603266 disk.drbd2266.in‚0) disk./.usagePercent‚16.74845859911694% disk.nvme0n1.out‚18769.8505859375 disk.sda.in‚ 0.05078125# memory_percent‚91.4689302444458) proc.memory.availableMB‚ 1397.73046875 disk.sda1.in‚0' disk.nvme0c0n1.out‚18769.8505859375 disk.sda2.out‚0 disk.dm-1.in‚ 0.05078125 disk.sda3.out‚ 6902.29296875 disk.sda2.in‚0 disk.nvme0n1.in‚ 334.1484375 disk.sda.out‚ 6902.29296875 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0$Mþ’É‚(:‡ ‘•ãÔ‰Ÿ©® gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚118.088' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.6176 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DÝÓàÊ‚(:ö ‘•ãÔžª—± disk.drbd1716.out‚0 disk.drbd2080.in‚0 proc.cpu.threads‚17 disk.rbd1.out‚10.42822265625 disk.sda.out‚ 6921.4296875 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 6921.4296875 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0) disk./.usagePercent‚16.74845859911694 disk.drbd2266.in‚0# disk.dm-0.out‚18770.13916015625 disk.sda3.in‚ 0.05078125$ memory_percent‚91.47047996520996( proc.memory.availableMB‚ 1397.4765625# proc.memory.rssMB‚ 2693.83984375 network.recv‚ 1.033041e+06 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda1.out‚0* proc.memory.percent‚16.441893577575684 disk.nvme0n1.in‚ 334.1484375 disk.sda2.out‚0 disk.dm-1.in‚ 0.05078125! disk.nvme0c0n1.in‚ 334.1484375 disk.dm-0.in‚ 334.1484375 cpu‚25.28607875374679% disk./.usageGB‚249.56413650512695 disk.rbd1.in‚0 network.sent‚ 7.756201e+06( disk.nvme0c0n1.out‚18770.13916015625 disk.rbd0.out‚0 disk.sda3.out‚ 6921.4296875& disk.nvme0n1.out‚18770.13916015625 disk.sda.in‚ 0.05078125eÚfò2Ë- _wandbruntime‚2857  _runtime‚2857¿ÞÇÌÁ & lr-AdamW/pg1‚0.0001913729989931222 & lr-AdamW/pg2‚0.0001913729989931222 trainer/global_step‚3549 % _timestamp‚1.7883982326302426e+09 _runtime‚2857.267084294 _step‚286êÈþÌÌÍÆ _runtime‚2857.267084294 _step‚286 & lr-AdamW/pg1‚0.0001913729989931222 & lr-AdamW/pg2‚0.0001913729989931222 trainer/global_step‚3549 % _timestamp‚1.7883982326302426e+09žùCÍ-2Î- _wandbruntime‚2857  _runtime‚2857‹ù±‚Ò'ÏÌO PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71410883 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20011753 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46297927 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚94363770018816 _step‚287 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚105122228496384 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚110199361943040 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46297927 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4990123240960 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117728849 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚105122204556288 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚99339532360457 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4989954241536 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788398232630938e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚99195903426560 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46297929 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚110199596468224 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27015554 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71410883 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚94363752979456 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚570 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47027307 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚104185857668096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 trainer/global_step‚3549 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚99195921429504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4990123240960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234525184 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4975762341641 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117708810 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5077367971840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚104186044670464 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46297929 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4989954241536 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27015556 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20012321 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210585088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71430920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚99195903426560 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚99195921429504 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71430920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4975597220241 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚99339350199697 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408URë×'ÐÑO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚105122204556288 XEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚99339532360457 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160760 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 4989954241536 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 $ _timestamp‚1.788398232630938e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020480 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002368 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚99195903426560 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24400 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚46297929 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚110199596468224 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚27015554 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚71410883 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚94363752979456 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24127 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚570 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47027307 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚104185857668096 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410711552 trainer/global_step‚3549 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚99195921429504 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 4990123240960 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234525184 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 4975762341641 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 117708810 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5077367971840 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚104186044670464 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46297929 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 4989954241536 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚27015556 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20012321 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210585088 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚71430920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚99195903426560 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999424 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚99195921429504 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚71430920 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 4975597220241 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24400 TADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚99339350199697 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚71410883 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20011753 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚46297927 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚94363770018816 _step‚287 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚105122228496384 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚568 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚110199361943040 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46297927 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 4990123240960 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20037 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 117728849 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160Ÿa3ÿ2Ñ- _wandbruntime‚2857  _runtime‚2857Ó^ñÁ> Ò×M ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4976986384785 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚104214707186688 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71450733 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚2857.959217744 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20017588 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4991347379712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117741256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚506 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27022933 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47040015 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚104214707186688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46310588 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚94389696161792 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46310586 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4991347379712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71450733 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚11022970)7R<Ÿ2119424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200086016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5078584614912 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 _step‚288 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚99223172791296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚99223190794240 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47040521 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5078784700928 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4977151506393 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71430670 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚105151141444608 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117761321 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224026112 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3549 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚104214520171008 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4991516392448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117761321 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46310588 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20017084 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7883982333218188e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117741256 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚110229926145536 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚99366864707545 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚94389713201152 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚99223172791296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46310586=â&ÓÜM YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚99223172791296 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46310586 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24400 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 4976986384785 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚104214707186688 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚71450733 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚2857.959217744 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20017588 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 4991347379712 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 117741256 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚506 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚27022933 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47040015 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚104214707186688 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚46310588 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚94389696161792 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46310586 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 4991347379712 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚71450733 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚110229702119424 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200086016 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5078584614912 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160968 _step‚288 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚99223172791296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚99223190794240 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚47040521 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5078784700928 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 4977151506393 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚71430670 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚105151141444608 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 117761321 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224026112 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚3549 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚104214520171008 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 4991516392448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 117761321 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46310588 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚20017084 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7883982333218188e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 117741256 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚110229926145536 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24400 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 VCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚99366864707545 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚94389713201152 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ýªTÑ2Ô- _wandbruntime‚2857  _runtime‚2857 ȾYÕÓ 1loss/SRqqt4/detection_loss‚1.0264380583287895 2loss/SRqqt2/assignment_loss‚2.0108620802822865 2loss/SRqqt3/assignment_loss‚1.7517609481894694 1loss/FBt4/assignment_loss‚0.20013163893211186 .loss/FRt1/detection_loss‚2.138984114516015 1loss/SRqqt1/detection_loss‚1.8085776359405157 0loss/FBt3/assignment_loss‚0.5193856026638638 _runtime‚2857.960611655 /loss/FRt4/assignment_loss‚2.215734088420868 2loss/SRqqt4/assignment_loss‚1.1739091058639255 /loss/FRt2/detection_loss‚2.7480223315102714 /loss/FBt3/detection_loss‚2.5121205286546187 /loss/FBt4/detection_loss‚1.2446597655466518 %loss/total_loss‚47.05055254354564 % _timestamp‚1.7883982333234751e+09 /loss/FRt2/assignment_loss‚4.203981603894915 1loss/SRqqt1/assignment_loss‚1.622025367856588 1loss/SRqqt3/detection_loss‚1.9832221259211384 trainer/global_step‚3549 0loss/FRt3/assignment_loss‚2.8450643122196198 0loss/FBt1/assignment_loss‚0.3084414255591683 /loss/FRt4/detection_loss‚2.0459220684387467 /loss/FBt1/detection_loss‚2.6716499262385898 _step‚289 .loss/FRt1/assignment_loss‚3.72965921607672 /loss/FRt3/detection_loss‚2.5219152227044104 /loss/FBt2/detection_loss‚2.9028206142512234 epoch‚2 /loss/FBt2/assignment_loss‚0.547045570385473 1loss/SRqqt2/detection_loss‚2.3182191911496615‘ •é^ÖØ 1loss/SRqqt4/detection_loss‚1.0264380583287895 2loss/SRqqt2/assignment_loss‚2.0108620802822865 2loss/SRqqt3/assignment_loss‚1.7517609481894694 1loss/FBt4/assignment_loss‚0.20013163893211186 .loss/FRt1/detection_loss‚2.138984114516015 1loss/SRqqt1/detection_loss‚1.8085776359405157 0loss/FBt3/assignment_loss‚0.5193856026638638 _runtime‚2857.960611655 /loss/FRt4/assignment_loss‚2.215734088420868 2loss/SRqqt4/assignment_loss‚1.1739091058639255 /loss/FRt2/detection_loss‚2.7480223315102714 /loss/FBt3/detection_loss‚2.5121205286546187 /loss/FBt4/detection_loss‚1.2446597655466518 %loss/total_loss‚47.05055254354564 % _timestamp‚1.7883982333234751e+09 /loss/FRt2/assignment_loss‚4.203981603894915 1loss/SRqqt1/assignment_loss‚1.622025367856588 1loss/SRqqt3/detection_loss‚1.9832221259211384 trainer/global_step‚3549 0loss/FRt3/assignment_loss‚2.8450643122196198 0loss/FBt1/assignment_loss‚0.3084414255591683 /loss/FRt4/detection_loss‚2.0459220684387467 /loss/FBt1/detection_loss‚2.6716499262385898 _step‚289 .loss/FRt1/assignment_loss‚3.72965921607672 /loss/FRt3/detection_loss‚2.5219152227044104 /loss/FBt2/detection_loss‚2.9028206142512234 epoch‚2 /loss/FBt2/assignment_loss‚0.547045570385473 1loss/SRqqt2/detection_loss‚2.3182191911496615¡Óa›†‘ׂ(:†  •ãÔ¹©­® gpu.0.gpu‚23.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚124.865' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.973 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0iOíX~Ø‚(:ó  •ãÔäœõ° disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.dm-0.in‚ 334.1484375 disk.rbd1.out‚10.42822265625' disk.nvme0c0n1.out‚18776.6298828125 disk.nvme0n1.in‚ 334.1484375% disk.nvme0n1.out‚18776.6298828125 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.05078125 disk.sda.out‚ 6951.25390625* disk./.usagePercent‚16.748854386764876 disk.sda1.out‚0% proc.memory.availableMB‚ 1397.1875 disk.rbd0.out‚0" disk.dm-0.out‚18776.6298828125! disk.nvme0c0n1.in‚ 334.1484375 disk.sda.in‚ 0.05078125* proc.memory.percent‚16.441893577575684 network.sent‚ 7.762114e+06 network.recv‚ 1.036787e+06 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2693.83984375 cpu‚25.28485820487192 proc.cpu.threads‚17 disk.dm-1.out‚ 6951.25390625 disk.sda2.in‚0 disk.dm-1.in‚ 0.05078125 disk.sda3.out‚ 6951.25390625$ memory_percent‚91.47224426269531$ disk./.usageGB‚249.5700340270996 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0ìa¿‚’Ù‚(:‡ ¯•ãԪδ® gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚111.883' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.3766 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105618997227520 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚100559371663360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 119354501 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72423622 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105619184229888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚105619184229888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚100559371663360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 % _timestamp‚1.7883982696693413e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚100706163295665 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46930877 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46930879 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47667148 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚100705981134905 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95660929135616 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 trainer/global_step‚3599 _runtime‚2894.306421121 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 119334462 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚100559389666304 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20283270 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5059625564160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5045234160049 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚526 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105618997227520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20282744 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95660912096256 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5059794563584 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0ì‚Þ'àØO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5045069038649 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚105618997227520 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020480 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚100559371663360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 119354501 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚528 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚72423622 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚105619184229888 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20037 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789965 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚105619184229888 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚100559371663360 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20039 % _timestamp‚1.7883982696693413e+09 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚100706163295665 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚46930877 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚46930879 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚47667148 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚100705981134905 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚95660929135616 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410711552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 trainer/global_step‚3599 _runtime‚2894.306421121 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 119334462 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚100559389666304 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚20283270 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5059625564160 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5045234160049 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚526 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002368 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚105618997227520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20282744 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚95660912096256 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121400 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24127 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5059794563584 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999424 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245253£PÑ2á- _wandbruntime‚2894  _runtime‚2894îU#ã&âÝM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 119386973 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚508 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚72443435 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119366908 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚510 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27391781 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410711552 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187015680 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚105647659730432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚100586641028096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169012736 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5061018702336 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚100733313481785 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72423372 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46943538 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204280320 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚2894.996507668 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119386973 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46943536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5061018702336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚3599 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119366908 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5046623324801 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46943536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47680215 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚100586641028096 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 % _timestamp‚1.7883982703589873e+09 _step‚292 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187015680 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚100733313481785 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚72423372 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚46943538 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169012736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204280320 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚2894.996507668 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 119386973 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚46943536 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5061018702336 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410711552 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚3599 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 119366908 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020480 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5046623324801 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚46943536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24127 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚47680215 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789965 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚100586641028096 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245253 % _timestamp‚1.7883982703589873e+09 _step‚292 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20065 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020480 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 119366908 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121608 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20063 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚510 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚27391781 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20063¤Š–û2ä- _wandbruntime‚2894  _runtime‚2894„ƒ [XåÒ 2loss/SRqqt4/assignment_loss‚1.4219025348083696 1loss/FBt4/assignment_loss‚0.29001849412953956 .loss/FBt1/detection_loss‚2.395670473575592 %loss/total_loss‚49.56738807323313 epoch‚2 .loss/FRt2/detection_loss‚2.913634222966653 /loss/FRt3/detection_loss‚2.8337714935050293 0loss/SRqqt4/detection_loss‚1.186128289371297 _step‚293 0loss/FRt1/assignment_loss‚4.1710509276390075 /loss/FRt3/assignment_loss‚3.540599228704677 /loss/FBt4/detection_loss‚0.6394197659504144 _runtime‚2894.997904424 0loss/FRt4/assignment_loss‚2.0351663048450765 /loss/FRt1/detection_loss‚2.1599050629138947 /loss/FRt4/detection_loss‚2.0141762705663075 /loss/FBt3/detection_loss‚2.5548404966081892 trainer/global_step‚3599 2loss/SRqqt3/assignment_loss‚1.3788504087161728 0loss/FBt2/assignment_loss‚0.6535175494524689 1loss/SRqqt1/detection_loss‚1.7716416437988696 0loss/SRqqt2/detection_loss‚2.476014566090372 -loss/FBt2/detection_loss‚2.85308800637722 2loss/SRqqt1/assignment_loss‚1.3833372570250346 0loss/FBt1/assignment_loss‚0.5177668360993266 1loss/SRqqt3/detection_loss‚2.1844039504847874 % _timestamp‚1.7883982703611171e+09 /loss/FRt2/assignment_loss‚5.117582224033497 2loss/SRqqt2/assignment_loss‚2.3420726267827883 0loss/FBt3/assignment_loss‚0.7328294387885502’‡³¨æ× 0loss/FBt3/assignment_loss‚0.7328294387885502 2loss/SRqqt4/assignment_loss‚1.4219025348083696 1loss/FBt4/assignment_loss‚0.29001849412953956 .loss/FBt1/detection_loss‚2.395670473575592 %loss/total_loss‚49.56738807323313 epoch‚2 .loss/FRt2/detection_loss‚2.913634222966653 /loss/FRt3/detection_loss‚2.8337714935050293 0loss/SRqqt4/detection_loss‚1.186128289371297 _step‚293 0loss/FRt1/assignment_loss‚4.1710509276390075 /loss/FRt3/assignment_loss‚3.540599228704677 /loss/FBt4/detection_loss‚0.6394197659504144 _runtime‚2894.997904424 0loss/FRt4/assignment_loss‚2.0351663048450765 /loss/FRt1/detection_loss‚2_n«iµ.1599050629138947 /loss/FRt4/detection_loss‚2.0141762705663075 /loss/FBt3/detection_loss‚2.5548404966081892 trainer/global_step‚3599 2loss/SRqqt3/assignment_loss‚1.3788504087161728 0loss/FBt2/assignment_loss‚0.6535175494524689 1loss/SRqqt1/detection_loss‚1.7716416437988696 0loss/SRqqt2/detection_loss‚2.476014566090372 -loss/FBt2/detection_loss‚2.85308800637722 2loss/SRqqt1/assignment_loss‚1.3833372570250346 0loss/FBt1/assignment_loss‚0.5177668360993266 1loss/SRqqt3/detection_loss‚2.1844039504847874 % _timestamp‚1.7883982703611171e+09 /loss/FRt2/assignment_loss‚5.117582224033497 2loss/SRqqt2/assignment_loss‚2.3420726267827883¥ºø=¤ç‚(:’ ¾•ãÔÐøª® gpu.0.gpu‚30.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚140.221' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.044200000000004 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0óhvè‚(:ô ¾•ãÔ¾ïØ° disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.availableMB‚ 1397.69140625 cpu‚25.282449236730205' disk.nvme0c0n1.out‚18777.6123046875 disk.sda.out‚ 7004.8828125 disk.drbd1716.out‚0 disk.sda3.in‚ 0.05078125 disk.sda.in‚ 0.05078125 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0# memory_percent‚91.4691686630249% disk./.usageGB‚249.57003784179688 disk.nvme0n1.in‚ 334.1484375 disk.dm-1.in‚ 0.05078125 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.rbd1.out‚10.51416015625! disk.nvme0c0n1.in‚ 334.1484375 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.dm-0.in‚ 334.1484375 network.sent‚ 7.875361e+06" disk.dm-0.out‚18777.6123046875# proc.memory.rssMB‚ 2693.83984375 proc.cpu.threads‚17% disk.nvme0n1.out‚18777.6123046875* proc.memory.percent‚16.441893577575684 network.recv‚ 1.046472e+06) disk./.usagePercent‚16.74885464277241 disk.rbd1.in‚0 disk.dm-1.out‚ 7004.8828125 disk.sda3.out‚ 7004.8828125 disk.sda2.out‚0 disk.drbd2265.in‚0¹gb‘é‚(:† Í•ãÔÎ÷Ž® gpu.0.gpu‚4.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚113.534' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7068 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àY敃ê‚(:ø Í•ãÔ§€„² disk.sda2.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚10.57666015625! disk.nvme0c0n1.in‚ 334.1484375( disk.nvme0c0n1.out‚18778.23486328125 network.recv‚ 1.053248e+06) disk./.usagePercent‚16.74885464277241 disk.drbd2265.out‚0 disk.sda.in‚ 0.05078125 disk.dm-1.out‚ 7038.0546875 proc.cpu.threads‚17% disk./.usageGB‚249.57003784179688 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.dm-0.in‚ 334.1484375 disk.drbd2266.in‚0 disk.drbd2266.out‚0$ memory_percent‚91.46974086761475 cpu‚25.281322609584958 disk.rbd0.in‚0# disk.dm-0.out‚18778.23486328125 disk.sda1.out‚0& disk.nvme0n1.out‚18778.23486328125) proc.memory.availableMB‚ 1397.59765625* proc.memory.percent‚16.441893577575684 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.05078125# proc.memory.rssMB‚ 2693.83984375 network.sent‚ 7.983899e+06 disk.drbd2080.out‚0 disk.sda3.out‚ 7038.0546875 disk.nvme0n1.in‚ 334.1484375 disk.sda.out‚ 7038.0546875 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.sda3.in‚ 0.05078125ï‡ bœë‚(:‘ Ü•ãÔùš® gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚150.486' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.097199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒH¢è‚ì‚(:÷ Ü•ãԱ﫱 disk.sda3.in‚ 0.05078125 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 7082.6171875 disk.drbd2266.out‚0$ memory_percent‚91.46974086761475 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.rbd1.out‚10.57666015625 disk.rbd0.in‚0 cpu‚25.297616617736743) disk./.usagePercent‚16.74885617881762) proc.memory.availableMB‚ 1397.59765625 network.sent‚ 7.98981e+06 disk.sda.out‚ 7082.6171875 disk.drbd2080.out‚0# disk.dm-0.out‚18778.37548828125 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.05078125 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 334.1484375 disk.nvme0n1.in‚ 334.1484375 network.recv‚ 1.057266e+06% disk./.usageGB‚249.57006072998047 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0* proc.memory.percent‚16.441893577575684 disk.sda.in‚ 0.05078125 disk.sda2.out‚0 disk.dm-0.in‚ 334.1484375 disk.sda3.out‚ 7082.6171875( disk.nvme0c0n1.out‚18778.37548828125# proc.memory.rssMB‚ 2693.83984375& disk.nvme0n1.out‚18778.37548828125™£·þ‘í‚(:† ë•ãÔц±® gpu.0.gpu‚4.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚122.154' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0TÖVe„î‚(:ù ë•ãÔ¬óµ± disk.rbd0.out‚0 disk.sda2.out‚0 disk.sda2.in‚0* proc.memory.percent‚16.497087478637695 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd1716.out‚0# memory_percent‚91.5250301361084 disk.rbd0.in‚0 disk.sda3.in‚ 0.05078125 disk.sda3.out‚ 7124.99609375 disk.drbd1716.in‚0 cpu‚25.323597215340737 disk.dm-1.in‚ 0.05078125 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.dm-0.in‚ 334.1484375# disk.dm-0.out‚18778.87353515625 disk.nvme0n1.in‚ 334.1484375& disk.nvme0n1.out‚18778.87353515625( disk.nvme0c0n1.out‚18778.87353515625 disk.sda.in‚ 0.05078125 disk.sda.out‚ 7124.99609375" proc.memory.rssMB‚ 2702.8828125 network.sent‚ 7.996053e+06 network.recv‚ 1.061955e+06% disk./.usageGB‚249.57004165649414 disk.dm-1.out‚ 7124.99609375 disk.drbd2266.in‚0! disk.nvme0c0n1.in‚ 334.1484375( proc.memory.availableMB‚ 1388.5390625* disk./.usagePercent‚16.748854898779946 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚10.60888671875I-ÈÞ‘ï‚(:† ú•ãÔš¤§® gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚122.154' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Lµ„ð‚(:ù ú•ãÔÜÇ„±& disk.nvme0n1.out‚18779.04150390625 disk.rbd0.out‚0 disk.sda1.out‚0& proc.memory.availableMB‚ 1397.59375 disk.nvme0n1.in‚ 334.1484375 disk.sda2.out‚0 network.sent‚ 8.001457e+06 disk.rbd1.out‚10.60888671875! disk.nvme0c0n1.in‚ 334.1484375 disk.sda2.in‚0 disk.dm-1.out‚ 7155.76171875 disk.drbd2080.in‚0$ memory_percent‚91.46976470947266 proc.cpu.threads‚17( disk.nvme0c0n1.out‚18779.04150390625 disk.sda.in‚ 0.05078125 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.05078125 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.sda3.in‚ 0.05078125 network.recv‚ 1.065622e+06 disk.drbd2265.out‚0 disk.rbd0.in‚0* proc.memory.percent‚16.441893577575684* disk./.usagePercent‚16.748854898779946 disk.rbd1.in‚0 disk.drbd2266.in‚0 disk.sda3.out‚ 7155.76171875# disk.dm-0.out‚18779.04150390625% disk./.usageGB‚249.57004165649414 disk.dm-0.in‚ 334.1484375# proc.memory.rssMB‚ 2693.83984375 disk.sda.out‚ 7155.76171875 disk.drbd2080.out‚0 disk.sda1.in‚0 cpu‚25.348569490562717M}¨ë2ñ- _wandbruntime‚2963  _runtime‚2963©›÷ @*òºT ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 4624220160 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚199 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15978 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚48582038 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 4912020480 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚102247307979776 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚281 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚108338013170688 hVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚ 5205288699904 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 5205181126656 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 YLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚73340535 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚107363110558720 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 410711552 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 cQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚ 5115929886720 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 4757245253 trainer/global_step‚3626 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 145310720 _MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚ 5115802578944 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 121150754 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2460 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 140964948 UHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚73340535 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15976 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚102247325982720 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 107573248 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚47810217 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 145310720 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚283 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 123925588 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 4912020480 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24127 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚102247325982720 UHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚73324559 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚27909462 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚113543170357248 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚217 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚107363255869440 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚26 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 5101184380941 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚27909460 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24400 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 121134776 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 127307776 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 410711552 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚416 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 417333248 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2460 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 5101060455353 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚73324559 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 5115929886720 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚48581755 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 4390226944 _runtime‚2963.965314995 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚47810217 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 406789965 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 417333248 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 5115802578944 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15976 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 5041553408 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚199 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚113543301870592 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 131513344 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚108337989230592 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚217 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 121134776 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 269877760 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15978 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚102365557786553 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚416 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚47810219 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚102365698751501 _step‚294 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚27909462 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚113543170357248 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚217 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚107363255869440 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚26 cQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.allocated‚ 5101184380941 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚27909460 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24400 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 121134776 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 127307776 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 410711552 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚416 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 417333248 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2460 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 _MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚ 5101060455353 QDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚73324559 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 5115929886720 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚48581755 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 4390226944 _runtime‚2963.965314995 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚47810217 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ZLDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.peak‚ 406789965 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 417333248 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 5115802578944 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15976 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 5041553408 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚199 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚113543301870592 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 131513344 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚108337989230592 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚217 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 121134776 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 269877760 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15978 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚102365557786553 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚416 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚47810219 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚102365698751501 _step‚294 FDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚982 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 269877760 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7883983402937658e+09 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚102266173165568 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 127307776 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 5116151714816 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24400 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚281 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 123925588 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 5116279022592 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 406789965 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15976 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 140964948 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚47820931 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚48593174 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚97282398884864 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚47820931 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 145310720 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2460 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚107382470191104 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚217 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24127 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 4912020480 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15978 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1003 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚982 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 269877760 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7883983402937658e+09 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚102266173165568 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 127307776 ZHDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚ 5116151714816 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24400 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚281 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 123925588 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 5116279022592 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 406789965 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15976 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 140964948 WJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚47820931 PCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚48593174 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚97282398884864 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚47820931 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 145310720 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2460 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚107382470191104 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚217 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24127 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 4912020480 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15978 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1003 B&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3672197461128235 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13735751807689667 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 _step‚296 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9878829121589661 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5152048468589783 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9523681998252869 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3402777910232544 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2606748938560486 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5211912989616394 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9584795236587524 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3624402582645416 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9725444316864014 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.47769588232040405 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_6_of_7‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5465799570083618 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 -jet/accuracy_4_of_4‚0.0020502645056694746 particle/accuracy_3_of_4‚1 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8859648704528809 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6622717380523682 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14506188035011292 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6515151262283325 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9774814248085022 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.38453876972198486 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.33981481194496155 particle/accuracy_1_of_6‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5037958025932312 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9704850316047668 % _timestamp‚1.7883983533071604e+09 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11017220467329025 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.002271009376272559 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4697834253311157 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 trainer/global_step‚3626 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735631346702576 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13111110031604767 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 particle/accuracy_1_of_7‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_6‚0.5384615659713745 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9815127849578857 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9940044283866882 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.49433523416519165 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3956426978111267 jet/accuracy_1_of_7‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14228050410747528 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,particle/sensitivity‚0.25076383352279663 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.45036470890045166 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3041214644908905 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8193426132202148 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.17327585816383362 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4033813178539276 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1350574642419815 particle/accuracy_2_of_2‚1 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.41974952816963196 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3333333730697632 + Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1324225813150406 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5307653546333313 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15640975534915924 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.31284719705581665 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1303534209728241 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.17531540989875793 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9916555285453796 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.13636364042758942 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.015864918008446693 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14439740777015686 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1369488388299942 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.34906938672065735 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9851523637771606 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.011383676901459694 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.35606062412261963 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1434357613325119 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5040361285209656 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.989964485168457 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.29590773582458496 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 & Purity/1FRt1SRqqtr£Ü)c4FBt/FRt_purity‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9688491821289062 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.027298850938677788 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 _runtime‚2977.946460751 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06245528906583786 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.43106135725975037 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15452168881893158 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.31588488817214966 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1349206417798996 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.024360213428735733 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 epoch‚2 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1496061235666275 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.20233586430549622 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06476229429244995 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4641484320163727 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1407565176486969 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24683907628059387 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_4_of_6‚1 particle/accuracy_4_of_7‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5737890005111694 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.006500377785414457 6 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.180630624294281 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4097222685813904 jet/accuracy_6_of_6‚0 *loss/val_total_loss‚48.793452807696454 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6818181872367859 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1388888955116272 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5061407685279846 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1407565176486969 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 (particle/accuracy‚0.8424374461174011 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17036661505699158 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4609951674938202 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9710884690284729 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4054597318172455 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.30658143758773804 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2062145322561264 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08772589266300201 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1411137878894806 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_3‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5175439119338989 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4583333432674408 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9765154719352722 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22248195111751556 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.15625 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9125296473503113 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 particle/accuracy_4_of_4‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18548600375652313 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.47067901492118835 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9959807991981506 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.41333332657814026 particle/accuracy_6_of_6‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.48273804783821106 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 *jet/accuracy_1_of_2‚0.7383725643157959 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.38758060336112976 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9773584604263306 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4443741738796234 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22399231791496277 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03207547217607498 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5027620196342468 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2242063283920288 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9984200596809387 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.19747473299503326 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_3_of_3‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.32499998807907104 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6350971460342407 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9288792610168457 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3548820912837982 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4183235764503479 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11920652538537979 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21088756620883942 particle/accuracy_1_of_4‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.023397749289870262 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9807692170143127 jet/accuracy_2_of_7‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19608205556869507 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9877618551254272 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_5‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.014197531156241894 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07673753052949905 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4838757812976837 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 'particle/f_score‚0.3318209946155548 5 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.17386394739151 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,jet/accuracy_3_of_3‚0.019955670461058617 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.24537038803100586 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16516338288784027 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11466679722070694 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_3_of_6‚0.1538461595773697 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.559374988079071 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9628472924232483 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05776454508304596 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.025512872263789177 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.43852856755256653 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14656192064285278 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9911994338035583 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.34129124879837036 particle/accuracy_4_of_5‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5908985733985901 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.43577584624290466 -jet/accuracy_4_of_5‚0.0026881720405071974 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19352297484874725 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4166666567325592 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18652474880218506 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9558308124542236 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2647046744823456 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.47380074858665466 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14547428488731384 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.25265756249427795 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4896751940250397 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.46258997917175293 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9837726950645447 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9519169926643372 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.34093302488327026 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3302469551563263 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.3987583518028259 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.021236559376120567 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4785620868206024 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4785620868206024 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.09603557735681534 particle/accuracy_1_of_5‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5739759802818298 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1388888955116272 jet/accuracy_5_of_7‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9693846106529236 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.4682347774505615 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 *jet/accuracy_1_of_3‚0.8475464582443237 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014109318144619465 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3892871141433716 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.49882692098617554 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29112404584884644 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.38045546412467957 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 particle/accuracy_2_of_7‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5075350403785706 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2842453420162201 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4766036868095398 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.995573878288269 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.004031282849609852 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3127976059913635 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20041930675506592 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5036624670028687 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9989926815032959 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4914644956588745 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5728021860122681 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.011678004637360573 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4100877344608307 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.49549952149391174 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09929078817367554 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 )jet/accuracy_2_of_4‚0.336047500371933 jet/accuracy_5_of_5‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4379471242427826 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_4_of_7‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9830245971679688 +jet/accuracy_3_of_5‚0.06529058516025543 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1567213088274002 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0223214291036129 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 particle/accuracy_2_of_6‚1 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.39893272519111633 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5666666626930237 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8993883728981018 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.085758276283741 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.48473793268203735 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13295625150203705 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3645833432674408 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.44525548815727234 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166666865348816 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9861018657684326 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3865740895271301 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 particle/accuracy_7_of_7‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9984200596809387 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 jet/accuracy_7_of_7‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.19552429020404816 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5129310488700867 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_4‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5096540451049805 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1485380083322525 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9890695810317993 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3798449635505676 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11213991045951843 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.38779953122138977 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5077653527259827 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_6‚0.057692307978868484 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0‰®©˜òûö¦ 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1041666716337204 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05776454508304596 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.025512872263789177 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.43852856755256653 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14656192064285278 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9911994338035583 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.34129124879837036 particle/accuracy_4_of_5‚1 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5908985733985901 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.43577584624290466 -jet/accuracy_4_of_5‚0.0026881720405071974 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19352297484874725 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4166666567325592 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.18652474880218506 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9558308124542236 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2647046744823456 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.21875 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.47380074858665466 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.14547428488731384 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.25265756249427795 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4896751940250397 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.46258997917175293 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9837726950645447 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 +particle/specificity‚0.9519169926643372 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.34093302488327026 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3302469551563263 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.3987583518028259 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.021236559376120567 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.4785620868206024 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.4785620868206024 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.09603557735681534 particle/accuracy_1_of_5‚1 +"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.25 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5739759802818298 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1388888955116272 jet/accuracy_5_of_7‚0 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9693846106529236 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_1‚0.4682347774505615 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 +"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.25 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 *jet/accuracy_1_of_3‚0.8475464582443237 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014109318144619465 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.3892871141433716 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.49882692098617554 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29112404584884644 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.38045546412467957 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 particle/accuracy_2_of_7‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5075350403785706 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2842453420162201 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4766036868095398 6 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.995573878288269 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.004031282849609852 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3127976059913635 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20041930675506592 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5036624670028687 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9989926815032959 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.4914644956588745 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5728021860122681 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.011678004637360573 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4100877344608307 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4954995214939117.Ö„Å‹v4 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09929078817367554 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 )jet/accuracy_2_of_4‚0.336047500371933 jet/accuracy_5_of_5‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4379471242427826 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 jet/accuracy_4_of_7‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9830245971679688 +jet/accuracy_3_of_5‚0.06529058516025543 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1567213088274002 9"Purity/2FRt0SRqqt1FBt/event_purity‚0.0223214291036129 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.4000000059604645 particle/accuracy_2_of_6‚1 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.39893272519111633 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5666666626930237 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 +"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.25 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_4‚0.8993883728981018 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 8"Purity/1FRt*SRqqt2FBt/event_purity‚0.085758276283741 :"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.48473793268203735 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13295625150203705 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3645833432674408 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.44525548815727234 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9166666865348816 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9861018657684326 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.3865740895271301 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 particle/accuracy_7_of_7‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 particle/accuracy_3_of_7‚1 ("Purity/*FRt2SRqqt3FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9984200596809387 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 jet/accuracy_5_of_6‚0 jet/accuracy_7_of_7‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.19552429020404816 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5129310488700867 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_4‚1 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5096540451049805 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1485380083322525 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9890695810317993 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3798449635505676 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.11213991045951843 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.38779953122138977 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5077653527259827 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ,jet/accuracy_4_of_6‚0.057692307978868484 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.34375 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.4000000059604645 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.4583333432674408 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18559500575065613 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 jet/accuracy_3_of_7‚0 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3263888955116272 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3829551637172699 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.45578286051750183 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0833333358168602 8"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.545634925365448 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +jet/accuracy_3_of_4‚0.05066046863794327 particle/accuracy_1_of_2‚1 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.43526747822761536 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4583333432674408 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.42105263471603394 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.23378950357437134 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4700660705566406 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.4360744059085846 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.2013888955116272 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.4375 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.31669190526008606 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.21456223726272583 particle/accuracy_2_of_3‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17976047098636627 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9766995906829834 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.43107616901397705 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_3_of_6‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.4583333432674408 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3672197461128235 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13735751807689667 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 _step‚296 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9878829121589661 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5152048468589783 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9523681998252869 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3402777910232544 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2606748938560486 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5211912989616394 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9584795236587524 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3624402582645416 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 & Purity/1FRt1SRqqt3FBt/FRt_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9725444316864014 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.47769588232040405 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_6_of_7‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5465799570083618 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 -jet/accuracy_4_of_4‚0.0020502645056694746 particle/accuracy_3_of_4‚1 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8859648704528809 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6622717380523682 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14506188035011292 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6515151262283325 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9774814248085022 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.38453876972198486 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.33981481194496155 particle/accuracy_1_of_6‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5037958025932312 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9704850316047668 % _timestamp‚1.7883983533071604e+09 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.11017220467329025 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.002271009376272559 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4697834253311157 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 trainer/global_step‚3626 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735631346702576 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.13111110031604767 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 particle/accuracy_1_of_7‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.949999988079071 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 *jet/accuracy_2_of_6‚0.5384615659713745 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9815127849578857 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9940044283866882 :"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.49433523416519165 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3956426978111267 jet/accuracy_1_of_7‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.14228050410747528 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,particle/sensitivity‚0.25076383352279663 8 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.45036470890045166 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3041214644908905 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8193426132202148 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.17327585816383362 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4033813178539276 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1350574642419815 particle/accuracy_2_of_2‚1 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.41974952816963196 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3333333730697632 + Purity/3FRt1SRqqt1FBt/FRt_purity‚0.3125 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1324225813150406 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5307653546333313 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.15640975534915924 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.31284719705581665 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1303534209728241 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.17531540989875793 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9916555285453796 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.13636364042758942 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.44999998807907104 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.015864918008446693 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14439740777015686 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1369488388299942 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.34906938672065735 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1041666716337204 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9851523637771606 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.011383676901459694 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.35606062412261963 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1434357613325119 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5040361285209656 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.989964485168457 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.29590773582458496 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9688491821289062 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.027298850938677788 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ."Purity/1FRt0SRqqt3FBt/event_purity‚0.09375 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 _runtime‚2977.946460751 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06245528906583786 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.43106135725975037 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.15452168881893158 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.31588488817214966 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1349206417798996 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.024360213428735733 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 epoch‚2 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1496061235666275 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.20233586430549622 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06476229429244995 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4641484320163727 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1407565176486969 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24683907628059387 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 particle/accuracy_4_of_6‚1 particle/accuracy_4_of_7‚1 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5737890005111694 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.006500377785414457 6 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.180630624294281 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4097222685813904 jet/accuracy_6_of_6‚0 *loss/val_total_loss‚48.793452807696454 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6818181872367859 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1388888955116272 ,"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.875 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5061407685279846 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1407565176486969 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 (particle/accuracy‚0.8424374461174011 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17036661505699158 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4609951674938202 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9710884690284729 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4054597318172455 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.30658143758773804 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2062145322561264 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08772589266300201 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1411137878894806 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 particle/accuracy_1_of_3‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5175439119338989 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4583333432674408 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 ("Purity/0FRt2SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9765154719352722 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22248195111751556 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.15625 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9125296473503113 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 particle/accuracy_4_of_4‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18548600375652313 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.47067901492118835 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9959807991981506 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.41333332657814026 particle/accuracy_6_of_6‚1 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 :"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.48273804783821106 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 *jet/accuracy_1_of_2‚0.7383725643157959 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.38758060336112976 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9773584604263306 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9333333373069763 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4443741738796234 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.22399231791496277 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03207547217607498 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5027620196342468 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2242063283920288 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9984200596809387 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.19747473299503326 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 particle/accuracy_3_of_3‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.32499998807907104 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6350971460342407 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9288792610168457 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3548820912837982 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4183235764503479 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.11920652538537979 ;"Purity/1FRt1SRqqt2FBt/event_purity‚0.010416666977107525 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21088756620883942 particle/accuracy_1_of_4‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.10526315867900848 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.023397749289870262 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9807692170143127 jet/accuracy_2_of_7‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19608205556869507 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9877618551254272 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_2_of_5‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.014197531156241894 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.07673753052949905 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.4838757812976837 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 'particle/f_score‚0.3318209946155548 5 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.17386394739151 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,jet/accuracy_3_of_3‚0.019955670461058617 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.24537038803100586 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16516338288784027 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11466679722070694 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_3_of_6‚0.1538461595773697 *"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.1666666716337204 & Purity/*FRt1SRqqt3FBt/FRt_purity‚0 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.559374988079071 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9628472924232483 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375¨‹/Èü‚(:’ ˜–ãÔÍÂõ­ gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚126.468' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.293599999999998 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹¤yyý‚(:î ˜–ãÔÕ¶‡± cpu‚25.383301677345923* disk./.usagePercent‚16.756951649065364 disk.sda.in‚ 0.05078125 disk.drbd1716.out‚0 disk.drbd2265.in‚0! proc.memory.rssMB‚ 2703.546875 disk.dm-0.in‚ 334.1484375 disk.rbd1.out‚ 10.671875$ memory_percent‚92.28060245513916% disk./.usageGB‚249.69068908691406 disk.rbd0.out‚0 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 334.1484375 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 334.1484375& disk.nvme0n1.out‚18902.63818359375 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.05078125 disk.dm-1.out‚ 7226.765625) proc.memory.availableMB‚ 1264.74609375* proc.memory.percent‚16.501140594482422 network.recv‚ 1.075996e+06 disk.rbd0.in‚0# disk.dm-0.out‚18902.63818359375 disk.dm-1.in‚ 0.05078125 proc.cpu.threads‚17 disk.sda2.out‚0 network.sent‚ 8.11547e+06 disk.sda3.out‚ 7226.765625( disk.nvme0c0n1.out‚18902.63818359375 disk.sda.out‚ 7226.765625 disk.drbd2080.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 m(Ò2þ- _wandbruntime‚2997  _runtime‚2997O7˜Èÿ $ _timestamp‚1.788398372458712e+09 _runtime‚2997.095609245 _step‚297 ' lr-AdamW/pg1‚0.00018939741202779597 ' lr-AdamW/pg2‚0.00018939741202779597 trainer/global_step‚3649ŽËZ­Í€Ç ' lr-AdamW/pg2‚0.00018939741202779597 trainer/global_step‚3649 $ _timestamp‚1.788398372458712e+09 _runtime‚2997.095609245 _step‚297 ' lr-AdamW/pg1‚0.00018939741202779597© ^m¤2- _wandbruntime‚2997  _runtime‚2997À Æ.‚ÒO ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5136457163598 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_ð£#¯ª&start/inactive_split.large_pool.freed‚28162754 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5151309436416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49021599 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73884208 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20858293 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 _runtime‚2997.096584787 _step‚298 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5241312993280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48238683 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73884208 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5151478436352 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚103222508418250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚108262656041472 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5151478436352 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48238683 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚550 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚109248524009472 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48238685 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚108262843044352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚103222326257486 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚108262656041472 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73864170 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚103111364608000 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚103111346605056 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5241521480704 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚103111364608000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚98085886133248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48238685 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚3649 $ _timestamp‚1.788398372459479e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚108262843044352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114490069430272 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114489837002752 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚109248547949568`¦:Ý'ƒ×O A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚49021599 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚73884208 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚20858293 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 _runtime‚2997.096584787 _step‚298 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5241312993280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚48238683 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚73884208 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5151478436352 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚103222508418250 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚108262656041472 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5151478436352 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48238683 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚550 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚109248524009472 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚48238685 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚108262843044352 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚103222326257486 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚108262656041472 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚73864170 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚103111364608000 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚103111346605056 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5241521480704 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚103111364608000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚98085886133248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚48238685 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚3649 $ _timestamp‚1.788398372459479e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚108262843044352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚114490069430272 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚114489837002752 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚109248547949568 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5136457163598 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28162754 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5151309436416 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969ª•ÇÆ{2„- _wandbruntime‚2997  _runtime‚2997•.¡Rã&…ÝM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20864191 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73883957 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚109277460897792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114520404170240 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48251344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚108291318544384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 _step‚299 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚103138633972736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122135299 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5242734798336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚103249840765338 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49034324 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122135299 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48251342 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚114520171756032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚2997.797279348 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122155365 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73883957 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚108291505560576 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122155365 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5152871587840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28170133 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5138011449754 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚103138615969792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5152702574592 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49033788 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73904021 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28170131 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 trainer/global_step‚3649 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚108291505560576 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚98111829315584 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5152702574592 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48251344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚98111812276224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5137846328142 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401ðÙOA †âM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48251342 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚73904021 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚103138615969792 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚103138633972736 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 122155365 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚73883957 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚108291505560576 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 122155365 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5152871587840 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28170133 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 \JDeviceStatsMo$— P§nitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5138011449754 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚103138615969792 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5152702574592 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49033788 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚73904021 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28170131 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 trainer/global_step‚3649 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚108291505560576 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚98111829315584 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5152702574592 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚48251344 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚98111812276224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5137846328142 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚20864191 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚73883957 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚109277460897792 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚114520404170240 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48251344 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚108291318544384 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 _step‚299 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚103138633972736 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 122135299 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5242734798336 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚103249840765338 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49034324 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 122135299 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48251342 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚114520171756032 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _runtime‚2997.797279348 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 123728505 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚99383045250048 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚3699 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48871633 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚109695982603776 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74876910 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5221149758976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74876910 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5220980759040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109695795600896 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚104474832844800 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74856872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚104589139353458 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123748545 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚104474814841856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110694171420672 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚74856872 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21125727 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚116006318621184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚104474814841856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5312351493632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚503 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0….þñø•×O UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 123728505 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚99383045250048 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 trainer/global_step‚3699 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚48871633 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚109695982603776 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚74876910 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5221149758976 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚74876910 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5220980759040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚109695795600896 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚104474832844800 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚74856872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚104589139353458 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 123748545 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚104474814841856 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚110694171420672 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚748568722&)å  QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21125727 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚116006318621184 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚104474814841856 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5312351493632 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚503 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _step‚302®€K´©2–- _wandbruntime‚3034  _runtime‚3034…Âtœã&—ÝM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48884294 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5222542910464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚508 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 trainer/global_step‚3699 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5313561956352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚109724645120000 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5207318146550 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28538981 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚116036885030912 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚109724458103808 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49670051 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚110723084368896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21131068 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49669541 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123760951 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚104502084206592 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74876659 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5222373897216 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74896723 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48884292 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚104616471700546 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48884292 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚104502084206592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74896723 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123781017 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚510 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚104616289539574 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5313776721920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚3034.545794154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚104502102209536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 % _timestamp‚1.7883984099085147e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74876659 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚116036646325248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚99408971393024Šaúè&˜âM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚116036646325248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚99408971393024 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚48884294 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5222542910464 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚508 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 trainer/global_step‚3699 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5313561956352 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚109724645120000 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5207318146550 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚28538981 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚116036885030912 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚109724458103808 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚49670051 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚110723084368896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21131068 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚49669541 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 123760951 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚104502084206592 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚74876659 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5222373897216 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚74896723 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚48884292 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚104616471700546 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚48884292 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚104502084206592 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚74896723 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 123781017 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚510 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚104616289539574 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5313776721920 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚3034.545794154 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚104502102209536 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 % _timestamp‚1.7883984099085147e+09 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚74876659 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416¯eç92™- _wandbruntime‚3034  _runtime‚3034{·ü\šÖ 1loss/SRqqt2/detection_loss‚2.5104698061943056 .loss/FBt1/detection_loss‚2.612350654602051 _runtime‚3034.546902416 0loss/FRt1/assignment_loss‚3.1901128945442347 /loss/FRt3/assignment_loss‚3.368561515302369 2loss/SRqqt2/assignment_loss‚1.7352962414423625 0loss/FBt1/assignment_loss‚0.5297638767055105 /loss/FBt3/detection_loss‚2.7768803000450135 % _timestamp‚1.7883984099101393e+09 /loss/FRt2/assignment_loss‚4.753929519653321 0loss/FRt4/assignment_loss‚1.7660433825324564 .loss/FRt3/detection_loss‚2.521736251133861 /loss/FBt2/detection_loss‚2.9305648108323417 0loss/FBt2/assignment_loss‚0.5521397280444944 /loss/FRt1/detection_loss‚2.0760420340184984 /loss/FRt2/detection_loss‚2.9325739085674285 1loss/SRqqt1/detection_loss‚1.7241040396469611 %loss/total_loss‚46.09307732723069 1loss/FBt4/assignment_loss‚0.24548047301064563 1loss/SRqqt3/detection_loss‚1.7423545929693407 epoch‚3 _step‚304 2loss/SRqqt4/assignment_loss‚1.2369822469468301 0loss/FBt3/assignment_loss‚0.2013235588092357 trainer/global_step‚3699 /loss/FBt4/detection_loss‚0.7438049574178747 2loss/SRqqt1/assignment_loss‚1.3956986188336655 2loss/SRqqt3/assignment_loss‚1.5556971330436007 /loss/FRt4/detection_loss‚1.7411540096589164 1loss/SRqqt4/detection_loss‚1.2500127732753754wv#[a›Û _runtime‚3034.546902416 0loss/FRt1/assignment_loss‚3.1901128945442347 /loss/FRt3/assignment_loss‚3.368561515302369 2loss/SRqqt2/assignment_loss‚1.7352962414423625 0loss/FBt1/assignment_loss‚0.5297638767055105 /loss/FBt3/detection_loss‚2.7768803000450135 % _timestamp‚1.7883984099101393e+09 /loss/FRt2/assignment_loss‚4.753929519653321 0loss/FRt4/assignment_loss‚1.7660433825324564 .loss/FRt3/detection_loss‚2.521736251133861 /loss/FBt2/detection_loss‚2.9305648108323417 0loss/FBt2/assignment_loss‚0.5521397280444944 /loss/FRt1/detection_loss‚2.0760420340184984 /loss/FRt2/detection_loss‚2.9325739085674285 1loss/SRqqt1/detection_loss‚1.7241040396469611 %loss/total_loss‚46.09307732723069 1loss/FBt4/assignment_loss‚0.24548047301064563 1loss/SRqqt3/detection_loss‚1.7423545929693407 epoch‚3 _step‚304 2loss/SRqqt4/assignment_loss‚1.2369822469468301 0loss/FBt3/assignment_loss‚0.2013235588092357 trainer/global_step‚3699 /loss/FBt4/detection_loss‚0.7438049574178747 2loss/SRqqt1/assignment_loss‚1.3956986188336655 2loss/SRqqt3/assignment_loss‚1.5556971330436007 /loss/FRt4/detection_loss‚1.7411540096589164 1loss/SRqqt4/detection_loss‚1.2500127732753754 1loss/SRqqt2/detection_loss‚2.5104698061943056 .loss/FBt1/detection_loss‚2.612350654602051°V«’œ‚(:‡ Ô–ãÔó® gpu.0.gpu‚18.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚106.922' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.3844 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0RÛ x‚(:í Ô–ãÔˆí²° disk.rbd0.out‚0 disk.sda1.in‚0# disk.nvme0n1.out‚19084.16796875 disk.drbd2266.in‚0# proc.memory.rssMB‚ 2604.08203125 disk.dm-1.in‚ 0.05078125 disk.drbd2080.in‚0! disk.nvme0c0n1.in‚ 334.1484375 disk.nvme0n1.in‚ 334.1484375 disk.drbd2080.out‚0 disk.sda3.in‚ 0.05078125 disk.rbd1.out‚10.88330078125 disk.drbd1716.out‚0 disk.dm-1.out‚ 7326.3359375 cpu‚25.37671739757693 disk.dm-0.out‚19084.16796875 disk.drbd2265.in‚0) proc.memory.availableMB‚ 1233.98046875 disk.sda2.out‚0* proc.memory.percent‚15.894055366516113% disk./.usageGB‚241.08332061767578 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.05078125 disk.sda2.in‚0 disk.drbd2265.out‚0( disk./.usagePercent‚16.1793039290316 disk.sda3.out‚ 7326.3359375% disk.nvme0c0n1.out‚19084.16796875 disk.sda.out‚ 7326.3359375$ memory_percent‚92.46838092803955 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.dm-0.in‚ 334.1484375 disk.drbd1716.in‚0 disk.drbd2266.out‚0 network.sent‚ 8.359849e+06 network.recv‚ 1.096396e+06àePž‚(:’ ã–ãÔ× ® gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚127.824' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.564799999999998 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_Wòø„Ÿ‚(:ù ã–ãÔ°·‘² disk.sda1.in‚0# proc.memory.rssMB‚ 2604.23828125 network.sent‚ 8.468681e+06% disk./.usageGB‚241.08337783813477 disk.drbd2080.in‚0 disk.sda3.in‚ 0.05078125! disk.nvme0c0n1.in‚ 334.1484375 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 334.1484375 disk.sda1.out‚0 disk.sda2.out‚0* disk./.usagePercent‚16.179307769144614 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.rbd1.in‚0# memory_percent‚92.4689769744873 proc.cpu.threads‚17 disk.sda3.out‚ 7338.14453125 disk.rbd1.out‚10.91552734375( disk.nvme0c0n1.out‚19195.18994140625 disk.sda.in‚ 0.05078125 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.05078125( proc.memory.availableMB‚ 1233.8828125 disk.rbd0.out‚0 disk.nvme0n1.in‚ 334.1484375 disk.dm-1.out‚ 7338.14453125 cpu‚25.375017996407532 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.sda.out‚ 7338.14453125) proc.memory.percent‚15.89500904083252& disk.nvme0n1.out‚19195.18994140625 network.recv‚ 1.103359e+06# disk.dm-0.out‚19195.18994140625Q)"q2 - _wandbruntime‚3070  _runtime‚3070HQFbÉ¡Ã % _timestamp‚1.7883984461768527e+09 _runtime‚3070.813725692 _step‚305 ' lr-AdamW/pg1‚0.00018735355384850178 ' lr-AdamW/pg2‚0.00018735355384850178 trainer/global_step‚3749Ê5º‘Î¢È ' lr-AdamW/pg1‚0.00018735355384850178 ' lr-AdamW/pg2‚0.00018735355384850178 trainer/global_step‚3749 % _timestamp‚1.7883984461768527e+09 _runtime‚3070.813725692 _step‚305±%^»;2£- _wandbruntime‚3070  _runtime‚3070Hæ>™ ¤ÕO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112139818831872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚100680187327488 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5275400800414 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 125354157 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75849574 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75869612 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117523055461888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 LFDeviceStatsMonitor.on_txâîBrain_batch_start/oversize_allocations.allocated‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112139842771968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49504585 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚111128935160320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105838283078656 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5290652081664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21393785 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28900454 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5383212689920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 trainer/global_step‚3749 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49504583 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5290821081600 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50294793 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚3070.814628456 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75869612 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21394337 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5275565921818 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 % _timestamp‚1.7883984461775672e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚111129122163200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49504585 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111128935160320 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50294239 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚111129122163200 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0Õ‹SFà'¥ÚO PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚75869612 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21394337 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5275565921818 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 % _timestamp‚1.7883984461775672e+09 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚111129122163200 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚49504585 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚111128935160320 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50294239 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚111129122163200 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚112139818831872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚100680187327488 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5275400800414 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 125354157 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚75849574 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚75869612 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚117523055461888 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚112139842771968 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚49504585 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚111128935160320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚105838283078656 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5290652081664 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21393785 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚28900454 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5383212689920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 trainer/global_step‚3749 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚49504583 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5290821081600 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50294793 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _runtime‚3070.814628456²iÌpö2¦- _wandbruntime‚3071  _runtime‚3071<˜€¡å&§ßM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5292045219840 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5276789964958 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50307011 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚111157784679424 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105865570446336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 trainer/global_step‚3749 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111157784679424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117553417366528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚111157597663232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75869361 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚542 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5384448977920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75889425 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28907833 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5292214233088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚105983102635754 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125406669 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21399722 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 _step‚307 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5276955086570 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5292045219840 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5292214233088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125386603 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21399180 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117553180758016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5384661646336 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7883984468731964e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125406669 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚105865552443392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚100706130509824 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112168731780096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚100706147549184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49517244 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7883984468731964e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 125406669 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚105865552443392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚100706130509824 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚544 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚112168731780096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚100706147549184 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚49517244 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50307011 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚111157784679424 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚105865570446336 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 trainer/global_step‚3749 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚111157784679424 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 41071206ø³?)F4 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚117553417366528 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚111157597663232 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚75869361 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚542 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5384448977920 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚75889425 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚28907833 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5292214233088 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚105983102635754 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 125406669 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21399722 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 _step‚307 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5276955086570 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5292045219840 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5292214233088 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 125386603 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21399180 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚117553180758016 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5384661646336 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚112562074719744 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚119039363848704 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76862314 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚113585466243072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50935311 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚112562261722624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚555 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚112562261722624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚107201751315456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5360323404288 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50137533 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚113585490183168 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚107322219063110 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚107201751315456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76842276 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5345037740226 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50137535 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚107201769318400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50137535 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50935868 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 trainer/global_step‚3799 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76842276 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚112562074719744 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5454110287360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126979809 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29269306 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21666007 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50137533 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5360323404288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 126999849 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5453897605632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _runtime‚3107.329348556 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _step‚310 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5360492404224 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚50137535 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚50935868 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 trainer/global_step‚3799 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚76842276 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚112562074719744 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5454110287360 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 126979809 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29269306 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21666007 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50137533 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5360323404288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 126999849 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5453897605632 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _runtime‚3107.329348556 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _step‚310 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5360492404224 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚112562074719744 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚119039363848704 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚76862314 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚113585466243072 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚50935311 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚112562261722624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚555 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚112562261722624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚107201751315456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5360323404288 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50137533 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚113585490183168 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚107322219063110 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚107201751315456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚76842276 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5345037740226 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50137535 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚107201769318400 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127012255 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚112590924238848 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚107229020680192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5361885555712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50948634 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _runtime‚ 3108.01328621 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127032321 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50150192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚112590737222656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5455528753152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 $ _timestamp‚1.788398483375933e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21671951 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119069693178880 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5346426904978 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21671422 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127032321 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50948103 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚107229038683136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5361885555712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5455313987584 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76882127 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76862063 trainer/global_step‚3799s‰ÔØè&ºâM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50150192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚107229038683136 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _runtime‚ 3108.01328621 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 127032321 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50150192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚112590737222656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5455528753152 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 $ _timestamp‚1.788398483375933e+09 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21671951 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚119069693178880 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5346426904978 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚21671422 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 127032321 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚50948103 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚107229038683136 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5361885555712 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5455313987584 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚76882127 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚76862063 trainer/global_step‚3799 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29276681 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚113614379191296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚107349551409990 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚102003306665984 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚119069931884544 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚112590924238848 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚107229020680192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29276683 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚76882127 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚102003289626624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 _step‚311 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50150194 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 127012255 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚112590924238848 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚107229020680192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5361885555712 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚50948634 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚498 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113995214279168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77855016 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5429994726912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128625501 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51573417 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113995401282048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚120555881525760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5414344437230 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77855016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29638156 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29638154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚108565237555200 _step‚314 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚115031113654272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50770483 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚108565219552256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199·S?uß'ÇÙO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚120556116050432 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5524978456064 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5429994726912 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚51573915 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚21935759 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚50770483 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚103274522600448 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 _runtime‚3143.797370139 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚50770485 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5430163726848 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5524767871488 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚498 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚113995214279168 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚77855016 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5429994726912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 128625501 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚51573417 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚113995401282048 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚120555881525760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5414344437230 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚77855016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚29638156 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚29638154 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚108565237555200 _step‚314 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚115031113654272 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚50770483 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚108565219552256 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚108565219552256 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚108688849998318 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 128605461 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚21935263 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096ºTqþ:2È- _wandbruntime‚3144  _runtime‚3144õŽÍ¾å&ÉßM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚114024063798272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5431556878336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77854765 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚315 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚115060050542592 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29645531 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚103300465782784 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7883985198453841e+09 _runtime‚3144.482929175 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50783142 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚108592488916992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5415733601774 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚120586446495744 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51586654 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚108592506919936 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21941121 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128637907 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77874829 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128637907 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128657973 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50783144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5431387865088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50783142 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚494 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚108716364506170 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128657973 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚108716182345198 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚3849 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5431556878336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29645533 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51586158 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚114023876782080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚114024063798272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚108592488916992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚120586214081536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5415898723386 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50783144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5526187479040 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199*4à65!ÊäM E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚120586214081536 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5415898723386 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚50783144 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5526187479040 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚114024063798272 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5431556878336 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚77854765 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚315 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚115060050542592 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚29645531 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚103300465782784 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7883985198453841e+09 _runtime‚3144.482929175 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚50783142 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚108592488916992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5415733601774 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚120586446495744 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚51586654 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚108592506919936 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚21941121 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 128637907 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚77874829 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 128637907 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 128657973 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚50783144 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5431387865088 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚50783142 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚494 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚108716364506170 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 128657973 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚108716182345198 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚3849 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5431556878336 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚29645533 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚51586158 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚114023876782080 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚114024063798272 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚108592488916992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2» gp2Ë- _wandbruntime‚3144  _runtime‚3144%^BPUÌÏ .loss/FRt3/detection_loss‚2.525906353443861 1loss/SRqqt4/detection_loss‚0.9982713148442667 epoch‚3 _runtime‚3144.484047596 %loss/total_loss‚49.84302052633162 .loss/FRt1/assignment_loss‚3.54194055745999 0loss/FRt4/assignment_loss‚1.4486944278081257 1loss/SRqqt1/assignment_loss‚1.979525039819154 1loss/SRqqt3/assignment_loss‚1.588153955861926 1loss/FBt3/assignment_loss‚0.17208015569485724 .loss/FRt1/detection_loss‚2.730017905123532 1loss/SRqqt3/detection_loss‚2.0762309795618057 /loss/FRt2/assignment_loss‚4.517448280538831 /loss/FBt2/assignment_loss‚0.596803125293346 .loss/FRt2/detection_loss‚3.106337398290634 1loss/SRqqt2/detection_loss‚2.9565764331817626 .loss/FBt3/detection_loss‚4.679058164358139 1loss/FBt4/assignment_loss‚0.26746625858546497 /loss/FRt4/detection_loss‚1.9966295442647404 /loss/FBt2/detection_loss‚2.5616576834158464 /loss/FBt4/detection_loss‚0.9163724430871715 trainer/global_step‚3849 /loss/FRt3/assignment_loss‚3.267368586361408 2loss/SRqqt2/assignment_loss‚1.7431108808517457 % _timestamp‚1.7883985198473008e+09 2loss/SRqqt4/assignment_loss‚1.2036142585548892 0loss/SRqqt1/detection_loss‚1.954953617399389 0loss/FBt1/assignment_loss‚0.6753510090864439 /loss/FBt1/detection_loss‚2.3394521534442902 _step‚316W°6ZÍÔ %loss/total_loss‚49.84302052633162 .loss/FRt1/assignment_loss‚3.54194055745999 0loss/FRt4/assignment_loss‚1.4486944278081257 1loss/SRqqt1/assignment_loss‚1.979525039819154 1loss/SRqqt3/assignment_loss‚1.588153955861926 1loss/FBt3/assignment_loss‚0.17208015569485724 .loss/FRt1/detection_loss‚2.730017905123532 1loss/SRqqt3/detection_loss‚2.0762309795618057 /loss/FRt2/assignment_loss‚4.517448280538831 /loss/FBt2/assignment_loss‚0.596803125293346 .loss/FRt2/detection_loss‚3.106337398290634 1loss/SRqqt2/detection_loss‚2.9565764331817626 .loss/FBt3/detection_loss‚4.679058164358139 1loss/FBt4/assignment_loss‚0.26746625858546497 /loss/FRt4/detection_loss‚1.9966295442647404 /loss/FBt2/detection_loss‚2.5616576834158464 /loss/FBt4/detection_loss‚0.9163724430871715 trainer/global_step‚3849 /loss/FRt3/assignment_loss‚3.267368586361408 2loss/SRqqt2/assignment_loss‚1.7431108808517457 % _timestamp‚1.7883985198473008e+09 2loss/SRqqt4/assignment_loss‚1.2036142585548892 0loss/SRqqt1/detection_loss‚1.954953617399389 0loss/FBt1/assignment_loss‚0.6753510090864439 /loss/FBt1/detection_loss‚2.3394521534442902 _step‚316 .loss/FRt3/detection_loss‚2.525906353443861 1loss/SRqqt4/detection_loss‚0.9982713148442667 epoch‚3 _runtime‚3144.484047596¼üA>’΂(:‡ ½—ãÔž³® gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚102.852' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚20.5704 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[cž0„Ï‚(:ù ½—ãÔÏ̉± disk.rbd0.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2604.23828125 disk.dm-1.out‚ 7434.12109375 disk.sda.in‚ 0.05078125 disk.sda2.out‚0 network.recv‚ 1.13172e+06 disk.drbd2266.out‚0 disk.rbd0.out‚0# disk.dm-0.out‚19255.81396484375 disk.sda3.in‚ 0.05078125 disk.rbd1.out‚11.04150390625! disk.nvme0c0n1.in‚ 334.1484375& disk.nvme0n1.out‚19255.81396484375 disk.drbd1716.in‚0) proc.memory.availableMB‚ 1233.01953125 cpu‚25.36574352431809 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 334.1484375 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 7434.12109375 disk.drbd1716.out‚0* disk./.usagePercent‚15.598104360467783% disk./.usageGB‚232.42302703857422 disk.dm-0.in‚ 334.1484375( disk.nvme0c0n1.out‚19255.81396484375 disk.sda.out‚ 7434.12109375 disk.sda2.in‚0 disk.dm-1.in‚ 0.05078125$ memory_percent‚92.47424602508545) proc.memory.percent‚15.89500904083252 network.sent‚ 8.708764e+06 disk.drbd2080.out‚0cwëЂ(:’ Ì—ãÔåÆ® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚138.866' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.773200000000003 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½•‹…Ñ‚(:ú Ì—ãÔ”èç° cpu‚25.364263202295113 proc.cpu.threads‚17 disk.dm-0.in‚ 334.1484375 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚19256.31787109375) proc.memory.availableMB‚ 1233.26171875# proc.memory.rssMB‚ 2604.23828125 disk.sda.out‚ 7442.36328125 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.05078125 disk.sda3.in‚ 0.05078125 network.sent‚ 8.816282e+06% disk./.usageGB‚232.42305374145508 disk.dm-1.out‚ 7442.36328125 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 7442.36328125 disk.rbd1.out‚11.04150390625 disk.drbd2266.out‚0 disk.sda1.out‚0$ memory_percent‚92.47276782989502 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0# disk.dm-0.out‚19256.31787109375 disk.nvme0n1.in‚ 334.1484375* disk./.usagePercent‚15.598106152520524! disk.nvme0c0n1.in‚ 334.1484375& disk.nvme0n1.out‚19256.31787109375 network.recv‚ 1.13691e+06 disk.sda.in‚ 0.05078125 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0) proc.memory.percent‚15.89500904083252TÞÒ‚(:’ Û—ãÔöý£® gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚127.287' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.457400000000003 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04ÀÕ{Ó‚(:ð Û—ãÔ”þ° disk.rbd0.out‚0* disk./.usagePercent‚15.598197035195179 disk.sda1.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2604.23828125) proc.memory.percent‚15.89500904083252 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.05078125 cpu‚25.36278947555905 disk.sda.in‚ 0.05078125 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.05078125 disk.rbd0.in‚0" disk.dm-0.out‚19266.2783203125$ memory_percent‚92.47307777404785% disk./.usageGB‚232.42440795898438 disk.dm-0.in‚ 334.15625% disk.nvme0n1.out‚19266.2783203125 disk.drbd2266.in‚0 disk.sda3.out‚ 7450.85546875 network.sent‚ 8.822669e+06 disk.drbd2265.in‚0 disk.drbd2080.in‚0( proc.memory.availableMB‚ 1233.2109375 disk.rbd1.out‚11.12744140625 disk.sda2.in‚0 disk.dm-1.out‚ 7450.85546875 disk.drbd2080.out‚0 disk.nvme0c0n1.in‚ 334.15625' disk.nvme0c0n1.out‚19266.2783203125 disk.nvme0n1.in‚ 334.15625 network.recv‚ 1.141699e+06 disk.sda.out‚ 7450.85546875 disk.drbd2266.out‚09k2Ô- _wandbruntime‚3180  _runtime‚31805¾ÖøÈÕ ' lr-AdamW/pg2‚0.00018416350001990953 trainer/global_step‚3899 $ _timestamp‚1.788398555395699e+09 _runtime‚3180.032532305 _step‚317 ' lr-AdamW/pg1‚0.00018416350001990953¤f$ÍÖÇ ' lr-AdamW/pg2‚0.00018416350001990953 trainer/global_step‚3899 $ _timestamp‚1.788398555395699e+09 _runtime‚3180.032532305 _step‚317 ' lr-AdamW/pg1‚0.00018416350001990953½MlòË2×- _wandbruntime‚3180  _runtime‚3180WC¨ˆÚ'ØÔO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚104571664677888 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5483816255638 _runtime‚3180.033455027 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚115428540841472 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30007006 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚110055480933526 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚110055663094290 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚116476785005568 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚115428540841472 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218973184 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51403433 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51403435 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5483981377042 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5499835049472 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130251153 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30007004 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 130231113 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚109928687789056 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22205498 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122072437074432 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78827680 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 $ _timestamp‚1.788398555396414e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115428353838592 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104571681717248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78847718 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚115428353838592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚109928687789056 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5499835049472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5499666049536 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚115428540841472 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218973184 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚51403433 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚51403435 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5483981377042 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5499835049472 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 130251153 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30007004 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 130231113 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚109928687789056 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22205498 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚122072437074432 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚78827680 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 $ _timestamp‚1.788398555396414e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚115428353838592 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚522 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚104571681717248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚78847718 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚115428353838592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚109928687789056 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5499835049472 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5499666049536 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78847467 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5501228200960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78847467 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5501059187712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5597085668352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚110082995441378 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122103004679168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22210545 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5501059187712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚104597624899584 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 K>DeviceStatsMonitor.on_train_†‹¸-batch_end/inactive_split.all.freed‚52224926 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 trainer/global_step‚3899 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130263559 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116505674013696 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130263559 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130283625 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109955957153792 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚115457203357696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064X¥Uåê&ÜäM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚122103004679168 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22210545 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5501059187712 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚104597624899584 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52224926 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 trainer/global_step‚3899 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 130263559 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚116505674013696 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 130263559 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 130283625 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚109955957153792 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚115457203357696 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚115457016341504 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚109955975156736 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5597306725376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚78847467 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5501228200960 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚78847467 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5501059187712 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5597085668352 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚110082995441378 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199¿4YÔ¯2Ý- _wandbruntime‚3180  _runtime‚3180;±~›WÞÑ 1loss/FBt3/assignment_loss‚0.28830190499623615 1loss/FBt4/assignment_loss‚0.15955110013730886 1loss/SRqqt1/detection_loss‚1.6744369665781658 % _timestamp‚1.7883985560881379e+09 0loss/FRt1/assignment_loss‚3.7495637384511658 0loss/FRt4/assignment_loss‚1.2214029243698827 0loss/SRqqt3/assignment_loss‚2.06931600566296 0loss/FBt1/assignment_loss‚1.0137263561288516 0loss/FBt2/assignment_loss‚0.2467269276643492 %loss/total_loss‚47.69522822084353 _runtime‚3180.724883808 2loss/SRqqt4/assignment_loss‚1.1008035085030965 /loss/FRt1/detection_loss‚2.0339256818012084 .loss/FRt4/detection_loss‚1.762311416643637 /loss/FBt3/detection_loss‚2.2750906869769096 epoch‚3 trainer/global_step‚3899 /loss/FRt3/assignment_loss‚3.926780743476672 2loss/SRqqt1/assignment_loss‚1.4300424500747964 1loss/SRqqt3/detection_loss‚1.8034531009547852 /loss/FBt4/detection_loss‚0.7339531326572255 _step‚320 1loss/SRqqt2/assignment_loss‚2.392293864680875 -loss/FRt2/detection_loss‚3.23793606814884 .loss/FRt3/detection_loss‚2.683374965802217 1loss/SRqqt2/detection_loss‚2.5296288190349454 /loss/FBt1/detection_loss‚3.0914713591337204 1loss/SRqqt4/detection_loss‚0.9878609673785311 .loss/FBt2/detection_loss‚2.822239542007446 /loss/FRt2/assignment_loss‚4.461035989579701éæ%9\ßÖ /loss/FRt3/assignment_loss‚3.926780743476672 2loss/SRqqt1/assignment_loss‚1.4300424500747964 1loss/SRqqt3/detection_loss‚1.8034531009547852 /loss/FBt4/detection_loss‚0.7339531326572255 _step‚320 1loss/SRqqt2/assignment_loss‚2.392293864680875 -loss/FRt2/detection_loss‚3.23793606814884 .loss/FRt3/detection_loss‚2.683374965802217 1loss/SRqqt2/detection_loss‚2.5296288190349454 /loss/FBt1/detection_loss‚3.0914713591337204 1loss/SRqqt4/detection_loss‚0.9878609673785311 .loss/FBt2/detection_loss‚2.822239542007446 /loss/FRt2/assignment_loss‚4.461035989579701 1loss/FBt3/assignment_loss‚0.28830190499623615 1loss/FBt4/assignment_loss‚0.15955110013730886 1loss/SRqqt1/detection_loss‚1.6744369665781658 % _timestamp‚1.7883985560881379e+09 0loss/FRt1/assignment_loss‚3.7495637384511658 0loss/FRt4/assignment_loss‚1.2214029243698827 0loss/SRqqt3/assignment_loss‚2.06931600566296 0loss/FBt1/assignment_loss‚1.0137263561288516 0loss/FBt2/assignment_loss‚0.2467269276643492 %loss/total_loss‚47.69522822084353 _runtime‚3180.724883808 2loss/SRqqt4/assignment_loss‚1.1008035085030965 /loss/FRt1/detection_loss‚2.0339256818012084 .loss/FRt4/detection_loss‚1.762311416643637 /loss/FBt3/detection_loss‚2.2750906869769096 epoch‚3 trainer/global_step‚3899À á#à‚(:’ ê—ãÔÈ—‚® gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚125.643' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.128600000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lêHwá‚(:ì ê—ãÔœ¯™° proc.cpu.threads‚17 disk.rbd1.out‚11.12744140625 disk.sda.in‚ 0.05078125 disk.drbd2266.out‚0 disk.drbd2265.out‚0# memory_percent‚92.4717664718628' disk.nvme0c0n1.out‚19266.5869140625 disk.dm-1.out‚ 7459.3515625 disk.sda3.out‚ 7459.3515625 disk.drbd1716.in‚0 network.recv‚ 1.145171e+06 disk.sda1.out‚0 disk.sda3.in‚ 0.05078125 disk.rbd1.in‚0 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.drbd2265.in‚0) proc.memory.availableMB‚ 1233.42578125 network.sent‚ 8.828142e+06$ disk./.usageGB‚232.4244384765625" disk.dm-0.out‚19266.5869140625 disk.nvme0c0n1.in‚ 334.15625 disk.sda.out‚ 7459.3515625# proc.memory.rssMB‚ 2604.23828125) proc.memory.percent‚15.89500904083252% disk.nvme0n1.out‚19266.5869140625 disk.dm-1.in‚ 0.05078125* disk./.usagePercent‚15.598199083255452 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 334.15625 disk.sda1.in‚0 disk.nvme0n1.in‚ 334.15625 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 cpu‚25.36134534899626.B'É‘â‚(:† ù—ãÔÔ‰˜® gpu.0.gpu‚36.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚139.155' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.831 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°“”~ã‚(:ó ù—ãԨČ² disk.nvme0c0n1.in‚ 334.15625 disk.nvme0n1.in‚ 334.15625 cpu‚25.359870637022706 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚92.47174263000488# proc.memory.rssMB‚ 2604.23828125 disk.sda.in‚ 0.05078125 disk.dm-1.in‚ 0.05078125 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 334.15625( disk.nvme0c0n1.out‚19270.05322265625 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0) proc.memory.percent‚15.89500904083252# disk.dm-0.out‚19270.05322265625 disk.sda3.in‚ 0.05078125 disk.sda3.out‚ 7467.23828125& disk.nvme0n1.out‚19270.05322265625) disk./.usagePercent‚15.59819652318011% disk./.usageGB‚232.42440032958984 disk.rbd1.in‚0( proc.memory.availableMB‚ 1233.4296875 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0 proc.cpu.threads‚17 network.sent‚ 8.936832e+06 network.recv‚ 1.151681e+06 disk.sda.out‚ 7467.23828125 disk.drbd1716.out‚0 disk.dm-1.out‚ 7467.23828125 disk.drbd2265.in‚0 disk.rbd1.out‚11.18994140625§cNÎ2ä- _wandbruntime‚3216  _runtime‚3216]Ç•ÿÇåÁ ' lr-AdamW/pg1‚0.00018306789397207275 ' lr-AdamW/pg2‚0.00018306789397207275 trainer/global_step‚3949 # _timestamp‚1.78839859197109e+09 _runtime‚3216.607968026 _step‚321û&ÇÌæÆ _step‚321 ' lr-AdamW/pg1‚0.00018306789397207275 ' lr-AdamW/pg2‚0.00018306789397207275 trainer/global_step‚3949 # _timestamp‚1.78839859197109e+09 _runtime‚3216.607968026ÁÓׄ2ç- _wandbruntime‚3216  _runtime‚3216u9š¥ˆ#èÔO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 131856765 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚105868840834048 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5553453195450 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚116861493398016 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5569506372096 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚52036385 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52036383 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚111292156025856 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚123588965844480 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚111422294029498 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚116861493398016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚111292156025856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52036383 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79840420 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30375856 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚123589200369152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚117922408476672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79840420 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131876805 trainer/global_step‚3949 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚116861680400896 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52036385 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30375854 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111292174028800 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5553288074046 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79820382 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111422111868734 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚105868823794688 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5666557367808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79820382 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5569337372160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52851033 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚111292174028800 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5666767952384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52851571 _step‚322 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMfžúqRonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚52851033 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚111292174028800 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5666767952384 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚52851571 _step‚322 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚116861493398016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚111292156025856 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52036383 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚79840420 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30375856 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚123589200369152 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚117922408476672 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚79840420 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 131876805 trainer/global_step‚3949 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚116861680400896 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52036385 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30375854 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚111292174028800 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5553288074046 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚79820382 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚111422111868734 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚105868823794688 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5666557367808 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚79820382Âëûè2ê- _wandbruntime‚3217  _runtime‚3217«¥h'ã&ëÝM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚116890342917120 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5570899523584 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚503 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5554677238590 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79860233 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5570899523584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚116890342917120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30383231 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 _step‚323 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111449444215614 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚105894766977024 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 trainer/global_step‚3949 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52864191 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚ 3217.2888738 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79840169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131909277 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚117951321424896 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚111319443393536 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚111319425390592 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5668185945088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5667975373824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79860233 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111449626376586 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚105894784016384 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131889211 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22480457 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚123619296798720 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52049042 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5570730510336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5554842360202 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52049042 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52863688 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131889211 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5570730510336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52049044 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131909277 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248ðËTè&ìâM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚117951321424896 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚111319443393536 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚111319425390592 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5668185945088 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5667975373824 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚79860233 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚111449626376586 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚105894784016384 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 131889211 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚22480457 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚123619296798720 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚52049042 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5570730510336 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5554842360202 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚52049042 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚52863688 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 131889211 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5570730510336 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52049044 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 131909277 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚116890342917120 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5570899523584 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚503 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5554677238590 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚79860233 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5570899523584 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚116890342917120 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚30383231 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 _step‚323 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚111449444215614 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚105894766977024 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 trainer/global_step‚3949 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚52864191 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚ 3217.2888738 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚79840169 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 131909277 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416Ãz*‘ò2í- _wandbruntime‚3217  _runtime‚3217Öº‡9VîÐ /loss/FRt1/assignment_loss‚3.529822082391807 .loss/FRt4/detection_loss‚1.987945988943631 _runtime‚3217.289903404 1loss/SRqqt2/assignment_loss‚2.241866790751616 2loss/SRqqt4/assignment_loss‚1.0731859701934834 .loss/FRt2/detection_loss‚2.920960237239969 % _timestamp‚1.7883985926531904e+09 0loss/FBt1/assignment_loss‚0.3684034623386521 /loss/FBt2/assignment_loss‚0.288335782976219 0loss/SRqqt2/detection_loss‚2.656257430712382 1loss/SRqqt3/detection_loss‚1.9838130594345562 .loss/FBt3/detection_loss‚2.737190435330073 /loss/FBt1/detection_loss‚2.7042436940329417 /loss/FBt4/detection_loss‚0.8581867400357118 epoch‚3 1loss/SRqqt1/assignment_loss‚1.719841553297426 /loss/FRt3/detection_loss‚2.4096324741840363 trainer/global_step‚3949 /loss/FRt1/detection_loss‚2.0157553111868247 0loss/SRqqt1/detection_loss‚1.716408016958407 /loss/FBt2/detection_loss‚4.1417970558007555 /loss/FRt2/assignment_loss‚4.153899517552606 2loss/SRqqt3/assignment_loss‚1.5669764737811005 0loss/FBt3/assignment_loss‚0.4836201233326089 1loss/FBt4/assignment_loss‚0.18568355920619853 _step‚324 0loss/FRt3/assignment_loss‚3.8519616696569656 0loss/FRt4/assignment_loss‚2.4716349907896737 1loss/SRqqt4/detection_loss‚1.2420758284943205 %loss/total_loss‚49.30949824862197à CïÕ 0loss/FBtTÒ¥dI1/assignment_loss‚0.3684034623386521 /loss/FBt2/assignment_loss‚0.288335782976219 0loss/SRqqt2/detection_loss‚2.656257430712382 1loss/SRqqt3/detection_loss‚1.9838130594345562 .loss/FBt3/detection_loss‚2.737190435330073 /loss/FBt1/detection_loss‚2.7042436940329417 /loss/FBt4/detection_loss‚0.8581867400357118 epoch‚3 1loss/SRqqt1/assignment_loss‚1.719841553297426 /loss/FRt3/detection_loss‚2.4096324741840363 trainer/global_step‚3949 /loss/FRt1/detection_loss‚2.0157553111868247 0loss/SRqqt1/detection_loss‚1.716408016958407 /loss/FBt2/detection_loss‚4.1417970558007555 /loss/FRt2/assignment_loss‚4.153899517552606 2loss/SRqqt3/assignment_loss‚1.5669764737811005 0loss/FBt3/assignment_loss‚0.4836201233326089 1loss/FBt4/assignment_loss‚0.18568355920619853 _step‚324 0loss/FRt3/assignment_loss‚3.8519616696569656 0loss/FRt4/assignment_loss‚2.4716349907896737 1loss/SRqqt4/detection_loss‚1.2420758284943205 %loss/total_loss‚49.30949824862197 /loss/FRt1/assignment_loss‚3.529822082391807 .loss/FRt4/detection_loss‚1.987945988943631 _runtime‚3217.289903404 1loss/SRqqt2/assignment_loss‚2.241866790751616 2loss/SRqqt4/assignment_loss‚1.0731859701934834 .loss/FRt2/detection_loss‚2.920960237239969 % _timestamp‚1.7883985926531904e+09ÄJ…/H‘ð‚(:† ˆ˜ãÔÚ犮 gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚124.865' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.973 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0g>9}ñ‚(:ò ˆ˜ãÔîŸø± disk.sda3.out‚ 7475.9296875 disk.dm-1.out‚ 7475.9296875 disk.drbd2265.out‚0 cpu‚25.358443139249456% disk./.usageGB‚232.42441940307617 disk.nvme0n1.in‚ 334.15625 disk.sda2.in‚0 disk.drbd2265.in‚0) proc.memory.availableMB‚ 1232.89453125) proc.memory.percent‚15.89500904083252 network.recv‚ 1.155969e+06 disk.rbd1.in‚0$ memory_percent‚92.47500896453857& disk.nvme0n1.out‚19270.39697265625 disk.drbd1716.in‚0* disk./.usagePercent‚15.598197803217781 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚19270.39697265625 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.nvme0c0n1.in‚ 334.15625 disk.sda.out‚ 7475.9296875 disk.sda2.out‚0 disk.dm-1.in‚ 0.05078125 disk.dm-0.in‚ 334.15625 disk.drbd1716.out‚0# disk.dm-0.out‚19270.39697265625 disk.rbd1.out‚11.18994140625 disk.drbd2266.in‚0 disk.drbd2266.out‚0# proc.memory.rssMB‚ 2604.23828125 proc.cpu.threads‚17 disk.sda.in‚ 0.05078125 disk.sda3.in‚ 0.05078125 network.sent‚ 8.942627e+06 disk.rbd0.out‚0Žæþ’ò‚(:‡ —˜ãÔéö–® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚137.263' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.4526 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”úY„ó‚(:ù —˜ãÔý–±# memory_percent‚92.4720287322998# proc.memory.rssMB‚ 2604.23828125 cpu‚25.357038286048464 disk.drbd2265.in‚0 disk.rbd1.in‚0 network.sent‚ 9.051629e+06* disk./.usagePercent‚15.665583850372977 disk.sda.out‚ 7483.4453125 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.nvme0n1.in‚ 334.18359375 network.recv‚ 1.162529e+06 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 7483.4453125 proc.cpu.threads‚17% disk./.usageGB‚233.42852020263672 disk.sda2.out‚0 disk.dm-1.in‚ 0.05078125 disk.dm-1.out‚ 7483.4453125 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 1233.3828125 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.sda3.in‚ 0.05078125" disk.nvme0c0n1.in‚ 334.18359375& disk.nvme0n1.out‚19271.41845703125) proc.memory.percent‚15.89500904083252 disk.drbd2080.out‚0 disk.dm-0.in‚ 334.18359375# disk.dm-0.out‚19271.41845703125( disk.nvme0c0n1.out‚19271.41845703125 disk.sda.in‚ 0.05078125 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.rbd1.out‚11.22216796875%Qäi2ô- _wandbruntime‚3252  _runtime‚3252‹>ó‚Éõà ' lr-AdamW/pg1‚0.00018195653829354737 ' lr-AdamW/pg2‚0.00018195653829354737 trainer/global_step‚3999 % _timestamp‚1.7883986282703023e+09 _runtime‚3252.907131563 _step‚3256uÎöÈ ' lr-AdamW/pg1‚0.00018195653829354737 ' lr-AdamW/pg2‚0.00018195653829354737 trainer/global_step‚3999 % _timestamp‚1.7883986282703023e+09 _runtime‚3252.907131563 _step‚325ÅQ&}#2÷- _wandbruntime‚3252  _runtime‚3252¥x ñÛ'øÕO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22744399 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52669335 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53489103 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚112655624262656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚107165982911488 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚119368055887872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚3999 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133502457 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30744706 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚107165999950848 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5622759892454 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5639177694720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5639008694784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚125105741411328 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚125105511080960 _runtime‚3252.907985746 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5639177694720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80833122 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80833122 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52669333 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚112788924964706 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22744939 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5737455193088 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5622925013858 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚52669333 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚80813084 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚112655624262656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230330368 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _step‚326 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚112655642265600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30744704 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚119368079827968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚112788742803942 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5737661583360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚118294632957440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _step‚326 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚112655642265600 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚30744704 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚119368079827968 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚112788742803942 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5737661583360 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚118294632957440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚22744399 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚52669335 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚53489103 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚112655624262656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚107165982911488 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚119368055887872 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚3999 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 133502457 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚30744706 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚107165999950848 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5622759892454 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5639177694720 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5639008694784 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚125105741411328 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚125105511080960 _runtime‚3252.907985746 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5639177694720 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚80833122 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚80833122 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚52669333 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚112788924964706 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚22744939ÆÙQO2ú- _wandbruntime‚3253  _runtime‚3253U 8tPûÞM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚118323482476544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚112682911630336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5640401832960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 trainer/global_step‚3999 _runtime‚3253.840451955 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52681994 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚125136074691072 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112682893627392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚107191943133184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80852935 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788398629203158e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22750611 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚119396968836096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚112816257311794 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5640570846208 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5640401832960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚119396992776192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚112682911630336 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53502156 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133534929 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚118323295460352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5624149056998 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.largK9›” e_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30752083 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133514863 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5624314178610 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52681994 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133534929 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217µä®Eé&üãM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5640401832960 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚119396992776192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚112682911630336 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚53502156 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 133534929 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚118323295460352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5624149056998 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚30752083 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 133514863 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5624314178610 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚52681994 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 133534929 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚118323482476544 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚112682911630336 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5640401832960 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 trainer/global_step‚3999 _runtime‚3253.840451955 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚52681994 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚125136074691072 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚112682893627392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚107191943133184 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚80852935 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788398629203158e+09 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚22750611 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚119396968836096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚112816257311794 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5640570846208Çø;U2ý- _wandbruntime‚3253  _runtime‚3253­%\YþÓ 0loss/FBt3/assignment_loss‚0.7546225065986315 /loss/FBt4/assignment_loss‚0.326545301414566 .loss/FRt2/detection_loss‚3.039304252465566 0loss/SRqqt1/detection_loss‚1.803433134880933 0loss/FBt2/assignment_loss‚0.7107339821068308 0loss/SRqqt3/detection_loss‚1.739836553254953 _runtime‚3253.841565918 /loss/FRt3/detection_loss‚2.6070361352629132 1loss/SRqqt2/detection_loss‚2.9127355924359075 /loss/FBt4/detection_loss‚0.8709105728962641 % _timestamp‚1.7883986292048264e+09 1loss/FBt1/assignment_loss‚0.42980654668062923 /loss/FRt1/detection_loss‚2.1099826991558075 /loss/FBt1/detection_loss‚1.8087652623653412 0loss/FRt3/assignment_loss‚3.3894351190990872 0loss/FRt4/assignment_loss‚1.8677946273927335 2loss/SRqqt3/assignment_loss‚1.3068315432621882 2loss/SRqqt4/assignment_loss‚1.3261218124459828 .loss/FRt4/detection_loss‚1.904094589529214 %loss/total_loss‚46.29158105419677 trainer/global_step‚3999 0loss/FRt1/assignment_loss‚2.9485339002175763 2loss/SRqqt2/assignment_loss‚1.8849249600122373 /loss/FBt3/detection_loss‚2.9296881357828775 /loss/FRt2/assignment_loss‚4.366220319271088 .loss/FBt2/detection_loss‚2.695551945612981 _step‚328 1loss/SRqqt4/detection_loss‚1.2071792657577223 epoch‚3 2loss/SRqqt1/assignment_loss‚1.3514922962947324%z«^ÿØ 1loss/SRqqt4/detection_loss‚1.2071792657577223 epoch‚3 2loss/SRqqt1/assignment_loss‚1.3514922962947324 0loss/FBt3/assignment_loss‚0.7546225065986315 /loss/FBt4/assignment_loss‚0.326545301414566 .loss/FRt2/detection_loss‚3.039304252465566 0loss/SRqqt1/detection_loss‚1.803433134880933 0loss/FBt2/assignment_loss‚0.7107339821068308 0loss/SRqqt3/detection_loss‚1.739836553254953 _runtime‚3253.841565918 /loss/FRt3/detection_loss‚2.6070361352629132 1loss/SRqqt2/detection_loss‚2.9127355924359075 /loss/FBt4/detection_loss‚0.8709105728962641 % _timestamp‚1.7883986292048264e+09 1loss/FBt1/assignment_loss‚0.42980654668062923 /loss/FRt1/detection_loss‚2.1099826991558075 /loss/FBt1/detection_loss‚1.8087652623653412 0loss/FRt3/assignment_loss‚3.3894351190990872 0loss/FRt4/assignment_loss‚1.8677946273927335 2loss/SRqqt3/assignment_loss‚1.3068315432621882 2loss/SRqqt4/assignment_loss‚1.3261218124459828 .loss/FRt4/detection_loss‚1.904094589529214 %loss/total_loss‚46.29158105419677 trainer/global_step‚3999 0loss/FRt1/assignment_loss‚2.9485339002175763 2loss/SRqqt2/assignment_loss‚1.8849249600122373 /loss/FBt3/detection_loss‚2.9296881357828775 /loss/FRt2/assignment_loss‚4.366220319271088 .loss/FBt2/detection_loss‚2.695551945612981 _step‚328È(ûÆþ’€‚(:‡ ¦˜ãÔ©„ÿ­ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚119.973' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9946 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;yB}‚(:ò ¦˜ãÔãÖ„± disk.rbd1.in‚0 disk.rbd1.out‚11.22216796875) proc.memory.availableMB‚ 1233.15234375 cpu‚25.35571289084306 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0) proc.memory.percent‚15.89500904083252 disk.dm-1.out‚ 7491.6953125 disk.dm-0.in‚ 337.2578125 disk.sda1.out‚0% disk./.usageGB‚233.75053787231445 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 7491.6953125 disk.sda.in‚ 0.05078125 proc.cpu.threads‚17 network.sent‚ 9.057219e+06! disk.nvme0c0n1.in‚ 337.2578125' disk.nvme0c0n1.out‚20629.4423828125 disk.drbd2265.out‚0 disk.sda2.out‚0 disk.drbd1716.out‚0# proc.memory.rssMB‚ 2604.23828125) disk./.usagePercent‚15.68719472637587" disk.dm-0.out‚20629.4423828125 disk.sda.out‚ 7491.6953125 disk.dm-1.in‚ 0.05078125 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.05078125 disk.nvme0n1.in‚ 337.2578125% disk.nvme0n1.out‚20629.4423828125# memory_percent‚92.4734354019165 network.recv‚ 1.166549e+06 disk.drbd1716.in‚0 disk.drbd2080.in‚0Žô_뜂‚(:‘ µ˜ãÔæ’î gpu.0.gpu‚2.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚119.444' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.888800000000003 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±£›Ãtƒ‚(:é µ˜ãÔ»¡±# disk.dm-0.out‚20631.92626953125 disk.sda2.out‚0$ memory_percent‚92.47331619262695# proc.memory.rssMB‚ 2604.23828125 disk.dm-0.in‚ 338.32421875 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0' proc.memory.availableMB‚ 1233.171875% disk./.usageGB‚233.75063705444336 disk.sda3.in‚ 0.05078125" disk.nvme0c0n1.in‚ 338.32421875 disk.drbd2265.in‚0) disk./.usagePercent‚15.68720138257176 disk.nvme0n1.in‚ 338.32421875& disk.nvme0n1.out‚20631.92626953125 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 11.28515625 disk.sda2.in‚0) proc.memory.percent‚15.89500904083252 network.recv‚ 1.172985e+06 disk.sda.out‚7497.125 disk.dm-1.out‚7497.125 proc.cpu.threads‚17 network.sent‚ 9.165763e+06 disk.dm-1.in‚ 0.05078125 cpu‚25.354241467078026 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.sda3.out‚7497.125( disk.nvme0c0n1.out‚20631.92626953125 disk.sda.in‚ 0.05078125 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0°¸’„‚(:‡ ĘãÔïï­ gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚120.799' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1598 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†U4Ò„…‚(:ù ĘãÔ‡éø° disk.rbd1.out‚ 11.28515625 disk.sda2.in‚0 proc.cpu.threads‚17 disk.rbd0.out‚0# disk.dm-0.out‚20632.26220703125 disk.sda1.out‚0 disk.drbd2266.in‚0# proc.memory.rssMB‚ 2604.23828125) proc.memory.percent‚15.89500904083252 cpu‚25.352792396110726 network.recv‚ 1.177296e+06 disk.dm-0.in‚ 338.32421875 disk.sda.out‚ 7504.65234375 disk.drbd1716.in‚0$ memory_percent‚92.47336387634277( proc.memory.availableMB‚ 1233.1640625 disk.sda3.out‚ 7504.65234375 disk.nvme0n1.in‚ 338.32421875 disk.sda.in‚ 0.05078125 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda3.in‚ 0.05078125& disk.nvme0n1.out‚20632.26220703125 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.drbd1716.out‚0* disk./.usagePercent‚15.687162981441624% disk./.usageGB‚233.75006484985352" disk.nvme0c0n1.in‚ 338.32421875( disk.nvme0c0n1.out‚20632.26220703125 disk.dm-1.in‚ 0.05078125 disk.dm-1.out‚ 7504.65234375 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.drbd2080.out‚0 network.sent‚ 9.17181e+06N–T2†- _wandbruntime‚3289  _runtime‚3289ªüɇà ' lr-AdamW/pg2‚0.00018082966458919105 trainer/global_step‚4049 % _timestamp‚1.7883986649676023e+09 _runtime‚3289.604493057 _step‚329 ' lr-AdamW/pg1‚0.00018082966458919105¶òû ÎˆÈ _runtime‚3289.604493057 _step‚329 ' lr-AdamW/pg1‚0.00018082966458919105 ' lr-AdamW/pg2‚0.00018082966458919105 trainer/global_step‚4049 % _timestamp‚1.7883986649676023e+09Édâï2‰- _wandbruntime‚3289  _runtime‚3289GìxÛ'ŠÕO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚81825824 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53302283 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31113556 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚114019092499456 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 _runtime‚3289.605369951 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚114019110502400 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚119727772516864 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5708849017344 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5708849017344 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚54129388 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚108463159067648 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚81825824 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚126621974385664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚563 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5692396832266 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119727772516864 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚114155555899914 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚119727959519744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5708680017408 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚114019110502400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚561 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135108069 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23015271 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53302283 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚4049 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31113554 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚114019110502400 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚561 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 135108069 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23015271 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53302283 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚4049 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31113554 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚563 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5692396832266 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚119727772516864 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚114155555899914 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚119727959519744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5708680017408 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚114182706086030 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31120931 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚108489085210624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚108489102249984 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5693785997018 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81825573 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚114046379867136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135140515 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135160581 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81845637 trainer/global_step‚4049 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54141757 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 % _timestamp‚1.7883986656536782e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54142283 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚114046361864192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5710242168832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135140515 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚126652543779840 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126652311365632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚114046379867136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120842616247296 _runtime‚3290.291053055 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5710073155584 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚119756622035968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5710242168832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120842640187392 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _step‚331 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53314942 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚119756622035968 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81825573 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119756435019776 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53314944 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81845637 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0Ü­|Cê&ŽäM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53314942 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5809903592448 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚114182706086030 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31120931 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚108489085210624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚108489102249984 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5693785997018 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚81825573 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚114046379867136 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 135140515 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 135160581 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚81845637 trainer/global_step‚4049 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54141757 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 % _timestamp‚1.7883986656536782e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚54142283 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚114046361864192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5710242168832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 135140515 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚126652543779840 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚126652311365632 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚114046379867136 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚120842616247296 _runtime‚3290.291053055 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5710073155584 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚119756622035968 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5710242168832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚120842640187392 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _step‚331 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚53314942 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚119756622035968 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚81825573 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚119756435019776 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53314944 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚81845637 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 135160581 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚114046361864192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5809695118336 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408Ëëˆ}2- _wandbruntime‚3290  _runtime‚3290éÖú&UÏ /loss/FRt3/assignment_loss‚3.282387517965757 0loss/SRqqt4/assignment_loss‚0.96828124385879 0loss/FBt1/assignment_loss‚0.4259183171064933 /loss/FRt4/detection_loss‚1.6795357977350553 _runtime‚3290.292173029 trainer/global_step‚4049 2loss/SRqqt2/assignment_loss‚1.9110380539839917 0loss/FBt3/assignment_loss‚0.5152631752192974 .loss/FBt2/detection_loss‚3.197298433099474 .loss/FRt1/assignment_loss‚3.88477555002485 2loss/SRqqt1/assignment_loss‚1.7529706832021474 /loss/FBt1/detection_loss‚2.9753034561872482 .loss/FBt4/detection_loss‚0.607761422423162 %loss/total_loss‚48.14968708561542 /loss/FRt2/assignment_loss‚4.123032667420127 0loss/FBt2/assignment_loss‚0.4882168835444775 /loss/FRt2/detection_loss‚2.6892056645769062 _step‚332 % _timestamp‚1.7883986656554244e+09 1loss/SRqqt3/assignment_loss‚1.755228031978563 1loss/FBt4/assignment_loss‚0.12102584315222395 1loss/SRqqt1/detection_loss‚2.4087981548574238 1loss/SRqqt2/detection_loss‚2.0779660140926186 .loss/FBt3/detection_loss‚3.100788176059723 1loss/SRqqt3/detection_loss‚1.9662234435478847 epoch‚3 0loss/FRt4/assignment_loss‚1.9140129844347635 /loss/FRt1/detection_loss‚2.7318009768213543 .loss/FRt3/detection_loss‚2.647172985168604 1loss/SRqqt4/detection_loss‚0.9256816091544834»§}ôZ‘Ô 1loss/SRqqt1/detection_loss‚2.4087981548574238 1loss/SRqqt2/detection_loss‚2.0779660140926186 .loss/FBt3/detection_loss‚3.100788176059723 1loss/SRqqt3/detection_loss‚1.9662234435478847 epoch‚3 0loss/FRt4/assignment_loss‚1.9140129844347635 /loss/FRt1/detection_loss‚2.7318009768213543 .loss/FRt3/detection_loss‚2.647172985168604 1loss/SRqqt4/detection_loss‚0.9256816091544834 /loss/FRt3/assignment_loss‚3.282387517965757 0loss/SRqqt4/assignment_loss‚0.96828124385879 0loss/FBt1/assignment_loss‚0.4259183171064933 /loss/FRt4/detection_loss‚1.6795357977350553 _runtime‚3290.292173029 trainer/global_step‚4049 2loss/SRqqt2/assignment_loss‚1.9110380539839917 0loss/FBt3/assignment_loss‚0.5152631752192974 .loss/FBt2/detection_loss‚3.197298433099474 .loss/FRt1/assignment_loss‚3.88477555002485 2loss/SRqqt1/assignment_loss‚1.7529706832021474 /loss/FBt1/detection_loss‚2.9753034561872482 .loss/FBt4/detection_loss‚0.607761422423162 %loss/total_loss‚48.14968708561542 /loss/FRt2/assignment_loss‚4.123032667420127 0loss/FBt2/assignment_loss‚0.4882168835444775 /loss/FRt2/detection_loss‚2.6892056645769062 _step‚332 % _timestamp‚1.7883986656554244e+09 1loss/SRqqt3/assignment_loss‚1.755228031978563 1loss/FBt4/assignment_loss‚0.12102584315222395̬Š:’‚(:’ Ó˜ãÔƒýˆ® gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚141.288' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.257600000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘£‡fz“‚(:ø Ó˜ãԤ㒱 disk.dm-1.in‚ 0.05078125 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 network.sent‚ 9.178052e+06 disk.sda.in‚ 0.05078125 disk.drbd1716.in‚0) proc.memory.percent‚15.89500904083252 disk.drbd2265.in‚0" disk.dm-0.out‚20633.4833984375$ memory_percent‚92.47190952301025) proc.memory.availableMB‚ 1233.40234375% disk./.usageGB‚233.75103378295898 disk.sda1.out‚0 disk.sda3.in‚ 0.05078125# proc.memory.rssMB‚ 2604.23828125" disk.nvme0c0n1.in‚ 338.32421875 disk.drbd1716.out‚0 disk.sda3.out‚ 7511.2421875 proc.cpu.threads‚17 disk.dm-1.out‚ 7511.2421875 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.rbd1.out‚11.34814453125 cpu‚25.351498150779957 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 338.32421875 network.recv‚ 1.181937e+06* disk./.usagePercent‚15.687228007355323' disk.nvme0c0n1.out‚20633.4833984375 disk.sda.out‚ 7511.2421875 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 338.32421875% disk.nvme0n1.out‚20633.4´µÀ 833984375QƒÏ‡”‚(:’ â˜ãԇŮ gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚128.213' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.642599999999998 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Y×°„•‚(:ù â˜ãÔúܘ° disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2604.23828125) proc.memory.percent‚15.89500904083252) disk./.usagePercent‚15.62871134122388 disk.sda.in‚ 0.05078125 disk.sda3.in‚ 0.05078125 disk.sda2.in‚0 disk.dm-1.in‚ 0.05078125 disk.drbd2266.in‚0 disk.sda3.out‚ 7520.6640625 disk.drbd1716.out‚0 disk.drbd2080.out‚0( disk.nvme0c0n1.out‚20645.01318359375 disk.nvme0n1.in‚ 338.32421875$ memory_percent‚92.47162342071533 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.dm-0.in‚ 338.32421875 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 338.32421875) proc.memory.availableMB‚ 1233.44921875 network.recv‚ 1.187113e+06% disk./.usageGB‚232.87909317016602 disk.sda2.out‚0 disk.drbd1716.in‚0& disk.nvme0n1.out‚20645.01318359375 proc.cpu.threads‚17 network.sent‚ 9.286024e+06 disk.dm-1.out‚ 7520.6640625 disk.drbd2265.in‚0# disk.dm-0.out‚20645.01318359375 disk.sda1.out‚0 disk.rbd1.out‚11.43408203125 cpu‚25.35015920526137 disk.sda.out‚ 7520.6640625 disk.drbd2266.out‚0J š;2–- _wandbruntime‚3325  _runtime‚3325‰}{Ç—Á _step‚333 ' lr-AdamW/pg1‚0.00017968750769779476 ' lr-AdamW/pg2‚0.00017968750769779476 trainer/global_step‚4099 # _timestamp‚1.78839870079352e+09 _runtime‚3325.430375884z@A1Ì˜Æ _runtime‚3325.430375884 _step‚333 ' lr-AdamW/pg1‚0.00017968750769779476 ' lr-AdamW/pg2‚0.00017968750769779476 trainer/global_step‚4099 # _timestamp‚1.78839870079352e+09ͯ¡€2™- _wandbruntime‚3325  _runtime‚3325;F,]Û'šÕO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 trainer/global_step‚4099 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚53935233 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚53935235 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 136733721 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚54768666 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5879143693824 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚334 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚53935235 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5778520339968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚109760301145088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚121161099079168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23286262 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚128138494404096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53935233 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82798488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206390272 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 _runtime‚3325.431310988 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5778520339968 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122259374650368 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5778351340032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23286789 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚115382578739200 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚529 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5879350084096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚115522186835122 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚121160912076288 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7883987007942166e+09 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5778351340032 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82818526 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚115522004674358 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚109760318184448 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚121161099079168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚121160912076288 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136753761 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚122259350710272 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚121160912076288 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 % _timestamp‚1.7883987007942166e+09 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5778351340032 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚82818526 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚115522004674358 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚109760318184448 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚121161099079168 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚121160912076288 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 136753761 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚122259350710272 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5778520339968 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚109760301145088 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚121161099079168 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23286262 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚128138494404096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚53935233 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚82798488 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206390272 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 _runtime‚3325.431310988 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5778520339968 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚122259374650368 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5778351340032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23286789 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚115382578739200 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚529 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5879350084096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚115522186835122 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064ÎC$i2œ- _wandbruntime‚3326  _runtime‚3326âïå&ßM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136786233 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53947894 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5779913491456 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136766167 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82838339 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚109786244327424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5763257815426 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54781347 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115409848103936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5779744478208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚128169055493120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5763092693814 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚128168823078912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5779913491456 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53947892 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚121189574579200 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚53947894 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31489783 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 136766167 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚115549337021238 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚109786261366784 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 trainer/global_step‚4099 % _timestamp‚1.7883987014579623e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 136786233 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚121189761595392 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚122288263658496 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 136786233 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚53947894 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 5779913491456 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 136766167 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚82838339 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚109786244327424 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5763257815426 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚54781347 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚115409848103936 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5779744478208 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚128169055493120 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5763092693814 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚128168823078912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5779913491456 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚53947892 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚121189574579200 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0Ï7S…#2Ÿ- _wandbruntime‚3326  _runtime‚3326U)2ÎY Ó /loss/FBt1/detection_loss‚2.3769548965824976 /loss/FBt3/detection_loss‚2.9428506890932717 trainer/global_step‚4099 0loss/FRt2/assignment_loss‚3.9797573666418753 2loss/SRqqt2/assignment_loss‚2.1650661260653763 1loss/SRqqt3/detection_loss‚2.1428239764645696 _runtime‚3326.096421443 2loss/SRqqt4/assignment_loss‚1.0897330390146145 1loss/SRqqt2/detection_loss‚2.5315124132694344 1loss/SRqqt1/detection_loss‚1.8634073954820634 epoch‚3 0loss/FBt1/assignment_loss‚0.5659120310241027 .loss/FRt1/detection_loss‚2.116556096336116 /loss/FRt4/detection_loss‚1.7812923270243186 /loss/FBt2/detection_loss‚3.0500756076404025 -loss/FBt4/detection_loss‚0.62733260674594 1loss/SRqqt3/assignment_loss‚1.707905733647446 0loss/FBt2/assignment_loss‚0.8000314877946428 1loss/FBt4/assignment_loss‚0.26544999252367285 .loss/FRt3/detection_loss‚2.330345171301261 1loss/SRqqt4/detection_loss‚1.0355368304711121 /loss/FRt3/assignment_loss‚3.504601334748061 0loss/FBt3/assignment_loss‚0.5046221849819025 %loss/total_loss‚47.22082004837026 % _timestamp‚1.7883987014596775e+09 /loss/FRt1/assignment_loss‚3.514755924110827 0loss/FRt4/assignment_loss‚1.9641506837473974 2loss/SRqqt1/assignment_loss‚1.4409308198094368 /loss/FRt2/detection_loss‚2.9192153138499104 _step‚336¼Q\?^¡Ø /loss/FBt1/detection_loss‚2.3769548965824976 /loss/FBt3/detection_loss‚2.9428506890932717 trainer/global_step‚4099 0loss/FRt2/assignment_loss‚3.9797573666418753 2loss/SRqqt2/assignment_loss‚2.1650661260653763 1loss/SRqqt3/detection_loss‚2.1428239764645696 _runtime‚3326.096421443 2loss/SRqqt4/assignment_loss‚1.0897330390146145 1loss/SRqqt2/detection_loss‚2.5315124132694344 1loss/SRqqt1/detection_loss‚1.8634073954820634 epoch‚3 0loss/FBt1/assignment_loss‚0.5659120310241027 .loss/FRt1/detection_loss‚2.116556096336116 /loss/FRt4/detection_loss‚1.7812923270243186 /loss/FBt2/detection_loss‚3.0500756076404025 -loss/FBt4/detection_loss‚0.62733260674594 1loss/SRqqt3/assignment_loss‚1.707905733647446 0loss/FBt2/assignment_loss‚0.8000314877946428 1loss/FBt4/assignment_loss‚0.26544999252367285 .loss/FRt3/detection_loss‚2.330345171301261 1loss/SRqqt4/detection_loss‚1.0355368304711121 /loss/FRt3/assignment_loss‚3.504601334748061 0loss/FBt3/assignment_loss‚0.5046221849819025 %loss/total_loss‚47.22082004837026 % _timestamp‚1.7883987014596775e+09 /loss/FRt1/assignment_loss‚3.514755924110827 0loss/FRt4/assignment_loss‚1.9641506837473974 2loss/SRqqt1/assignment_loss‚1.4409308198094368 /loss/FRt2/detection_loss‚2.9192153138499104 _step‚336ÐÈþ‘’¢‚(:‡ ñ˜ãÔ¿ç­® gpu.0.gpu‚36.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚35.0 gpu.0.powerWatts‚144.824' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.9648 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Nîsø‡£‚(:ü ñ˜ãÔ„ª™²& disk.nvme0n1.out‚20670.52001953125 disk.sda3.out‚ 7529.08984375 disk.rbd1.out‚11.58251953125 disk.sda1.in‚0) proc.memory.availableMB‚ 1233.19921875# disk./.usageGB‚232.879150390625 disk.dm-1.out‚ 7529.08984375 disk.drbd2266.out‚0 disk.nvme0n1.in‚ 338.32421875 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0" disk.nvme0c0n1.in‚ 338.32421875( disk.nvme0c0n1.out‚20670.52001953125 disk.sda2.out‚0 disk.dm-1.in‚ 0.05078125 disk.drbd2080.in‚0 disk.sda1.out‚0) proc.memory.percent‚15.89500904083252 network.recv‚ 1.191379e+06* disk./.usagePercent‚15.628715181336894 disk.rbd1.in‚0 disk.rbd0.out‚0$ memory_percent‚92.47314929962158 cpu‚25.348785793290205 proc.cpu.threads‚17 disk.sda.in‚ 0.05078125 disk.rbd0.in‚0 disk.dm-0.in‚ 338.32421875# proc.memory.rssMB‚ 2604.23828125 disk.sda.out‚ 7529.08984375 network.sent‚ 9.291905e+06 disk.sda3.in‚ 0.05078125 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0# disk.dm-0.out‚20670.52001953125] s’¤‚(:‡ €™ãÔðÏð­ gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚122.296' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4592 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ž*=󄥂(:ù €™ãÔœ·ï¯ disk.drbd1716.out‚0 disk.dm-1.out‚ 7536.0078125 disk.drbd2080.in‚0 disk.dm-0.in‚ 338.32421875 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2604.23828125 cpu‚25.34749221753921 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 7536.0078125) proc.memory.percent‚15.89500904083252 disk.sda3.in‚ 0.05078125 disk.rbd1.out‚11.81689453125 disk.sda2.out‚0 disk.drbd2266.in‚0# disk.dm-0.out‚20670.69580078125" disk.nvme0c0n1.in‚ 338.32421875$ memory_percent‚92.47133731842041) disk./.usagePercent‚15.62861405836087% disk./.usageGB‚232.87764358520508 disk.sda.in‚ 0.05078125 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.sda1.out‚0& disk.nvme0n1.out‚20670.69580078125) proc.memory.availableMB‚ 1233.49609375 disk.sda.out‚ 7536.0078125 disk.dm-1.in‚ 0.05078125 disk.nvme0n1.in‚ 338.32421875 network.sent‚ 9.400545e+06 disk.drbd1716.in‚0 disk.drbd2265.out‚0( disk.nvme0c0n1.out‚20670.69580078125 network.recv‚ 1.197374e+06 disk.sda2.in‚0Ø—?h’¦‚(:‡ ™ãÔ¼»³® gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚138.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.6658 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚01(ìÖ„§‚(:ù ™ãÔÃÑê° disk.sda3.out‚ 7537.15234375# disk.dm-0.out‚20671.23486328125 disk.sda1.out‚0& disk.nvme0n1.out‚20671.23486328125 disk.sda2.out‚0 disk.drbd1716.in‚0$ memory_percent‚92.47260093688965( proc.memory.availableMB‚ 1233.2890625 disk.drbd1716.out‚0 network.recv‚ 1.202026e+06 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd1.out‚11.87939453125" disk.nvme0c0n1.in‚ 338.32421875# proc.memory.rssMB‚ 2604.23828125 cpu‚25.34615882078251* disk./.usagePercent‚15.622646266730195 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 7537.15234375 disk.dm-1.in‚ 0.05078125 disk.drbd2266.in‚0 disk.drbd2266.out‚0) proc.memory.percent‚15.89500904083252 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 338.32421875 network.sent‚ 9.4068e+06 proc.cpu.threads‚17 disk.sda2.in‚0$ disk./.usageGB‚232.7887191772461 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.sda3.in‚ 0.05078125( disk.nvme0c0n1.out‚20671.23486328125 disk.nvme0n1.in‚ 338.32421875 disk.dm-1.out‚ 7537.15234375 disk.sda.in‚ 0.05078125U ‹P2¨- _wandbruntime‚3360  _runtime‚3360Dv—É©Ã _runtime‚3360.950038095 _step‚337 ' lr-AdamW/pg1‚0.00017853030564314278 ' lr-AdamW/pg2‚0.00017853030564314278 trainer/global_step‚4149 % _timestamp‚1.7883987363131971e+09=g´~ÎªÈ ' lr-AdamW/pg1‚0.00017853030564314278 ' lr-AdamW/pg2‚0.00017853030564314278 trainer/global_step‚4149 % _timestamp‚1.7883987363131971e+09 _runtime‚3360.950038095 _step‚337Ñ!~2«- _wandbruntime‚3360  _runtime‚3360‡“¸Û'¬ÕO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83791190 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚122594238638592 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23558412 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83811228 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚123704998121472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5831340469082 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚123705022061568 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54568185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54568183 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚116746046976000 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 _runtime‚3360.950860627 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚116746046976000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55409666 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5848022662656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83811228 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5831175347678 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4149 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚116746028973056 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54568185 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31851254 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5949959038976 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚122594051635712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55410223 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138359373 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚555 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54568183 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚111057477301248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23558967 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚116888635609566 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5848191662592 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚557 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31851256 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5950169623552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚116888817770330 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚116746028973056 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚122594051635712 _step‚338 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5848022662656‚$£[­ÚO E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚116746028973056 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚122594051635712 _step‚338 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5848022662656 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚83791190 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚122594238638592 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23558412 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚83811228 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚123704998121472 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5831340469082 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚123705022061568 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚54568185 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 4173§Ü«3…33248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚54568183 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚116746046976000 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 _runtime‚3360.950860627 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚116746046976000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚55409666 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5848022662656 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚83811228 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5831175347678 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚4149 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚116746028973056 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚54568185 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚31851254 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 5949959038976 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚122594051635712 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚55410223 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 138359373 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚555 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚54568183 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚111057477301248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚23558967 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚116888635609566 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5848191662592 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚557 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚31851256 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 5950169623552 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚116888817770330 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0ÒmìÙ×2®- _wandbruntime‚3361  _runtime‚3361È÷ä&¯ÞM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31858631 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55422934 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _runtime‚3361.624261658 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55422399 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111083420483584 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚111083403444224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138411885 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚116773316340736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54580842 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129685522878464 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138411885 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83831041 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚122622901154816 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5832729633834 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 $ _timestamp‚1.788398736986977e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138391819 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚116773298337792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5849415800832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5951587868672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54580844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚116916150117418 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5849415800832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚123733935009792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5832564512222 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5951377297408 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚4149 _step‚339 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23563768 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚122622901154816 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚116773316340736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31858633 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83810977 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217p’êé&°ãM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 138391819 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚83810977 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚54580844 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚116916150117418 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5849415800832 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚123733935009792 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5832564512222 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 5951377297408 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚4149 _step‚339 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚23563768 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚122622901154816 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚116773316340736 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚31858633 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚83810977 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚31858631 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚55422934 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _runtime‚3361.624261658 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚55422399 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚111083420483584 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚111083403444224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 138411885 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚116773316340736 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚54580842 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚129685522878464 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 138411885 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚83831041 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚122622901154816 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5832729633834 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 $ _timestamp‚1.788398736986977e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 138391819 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚116773298337792 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5849415800832 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 5951587868672 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84803930 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6020757799936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5917862985216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5917862985216 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚124027378198016 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55201135 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚131171403332608 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84783892 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _runtime‚3396.825763455 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚118109497209856 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56049072 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚118255266544774 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125150645532672 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚124027378198016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 $ _timestamp‚1.788398772188761e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚507 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140005065 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚125150669472768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5917693985280 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚131171635760128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚124027191195136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6020966287360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112354636418048 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5900647166086 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚124027191195136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55201133 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84803930 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 139985025 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5917693985280 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚118109515212800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56049579 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚342 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚118109497209856cÿƨß'½ÙO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 139985025 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5917693985280 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚118109515212800 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56049579 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚342 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚118109497209856 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32220104 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚23828968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 5900812287490 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚55201133 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32220106 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚118109515212800 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚84803930 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6020757799936 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5917862985216 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5917862985216 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚124027378198016 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚55201135 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚131171403332608 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚84783892 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _runtime‚3396.825763455 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚118109497209856 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56049072 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚118255266544774 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚125150645532672 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚124027378198016 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 $ _timestamp‚1.788398772188761e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚507 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 140005065 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚125150669472768 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5917693985280 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚131171635760128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚124027191195136 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6020966287360 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚112354636418048 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 5900647166086 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚124027191195136 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚55201133 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚84803930 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199Ö›«é2¾- _wandbruntime‚3397  _runtime‚3397^9`5å&¿ßM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚124056040714240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5919256136704 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23834966 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55213792 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56061923 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6022386143744 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84823743 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84803679 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140017471 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84803679 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚131201736150528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5919087123456 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55213794 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5902201452242 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140017471 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 _step‚343 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140037537 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096’  ÀäM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 5919087123456 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 140017471 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚84803679 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚131201736150528 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 5919087123456 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚55213794 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5902201452242 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 140017471 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 _step‚343 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 140037537 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚124056040714240 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5919256136704 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚23834966 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚55213792 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56061923 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6022386143744 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚84823743 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚84803679 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚125460330754560 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚119472965446656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85776594 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32588954 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24101503 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85796632 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6091853029888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32588956 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚113651778495488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 % _timestamp‚1.7883988077179508e+09 _runtime‚3432.354981251 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚119472983449600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5987534307840 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5987534307840 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24100973 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚125460517757440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56689927 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚119622079640746 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119621897479982 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85776594 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56690459 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚126596316883968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚119472983449600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚125460330754560 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5987365307904 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 _step‚346 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚55834085 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚119472965446656 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 141630717 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚113651795534848 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚132687929097728 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5987365307904 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚132687929097728 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 5987365307904 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚125460330754560 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚119472965446656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚85776594 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚32588954 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24101503 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚85796632 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6091853029888 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32588956 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚113651778495488 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 % _timestamp‚1.7883988077179508e+09 _runtime‚3432.354981251 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚119472983449600 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 5987534307840 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 5987534307840 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24100973 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚125460517757440 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚56689927 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚119622079640746 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚119621897479982 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚85776594 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚56690459 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚98xƒPIö2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚126596316883968 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚119472983449600 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚125460330754560 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 5987365307904 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 _step‚346Úäe¹†2Î- _wandbruntime‚3433  _runtime‚3433µCOå&ÏßM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚125489180273664 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56702724 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119649229826862 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5971508149038 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6093266036736 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5971673270650 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚119500252814336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32596331 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85796381 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56703247 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132718261357568 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119649411987834 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141663189 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141643123 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚132718495868928 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚3433.026412698 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85816445 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5988927459328 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32596333 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚113677738717184 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚126625229832192 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141663189 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24106914 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85796381 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚113677721677824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚119500234811392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚347Uæ„ê&ÐäM FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚56702724 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚119649229826862 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 5971508149038 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6093266036736 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 5971673270650 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚119500252814336 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32596331 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚85796381 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚56703247 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚132718261357568 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚119649411987834 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 141663189 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 141643123 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚132718495868928 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚3433.026412698 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚85816445 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 5988927459328 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32596333 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚113677738717184 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚126625229832192 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 141663189 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24106914 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚85796381 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚113677721677824 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚119500234811392 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚347 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚125489180273664Û|‹¥2Ñ- _wandbruntime‚3433  _runtime‚3433`kÚ/YÒÓ /loss/FBt1/detection_loss‚2.2926499359309673 0loss/FRt3/assignment_loss‚4.1166813232396775 2loss/SRqqt4/assignment_loss‚0.8939145084953418 0loss/FBt1/assignment_loss‚0.5824865751783364 0loss/FBt3/assignment_loss‚0.5091993389651179 1loss/SRqqt2/detection_loss‚2.2230871549674442 .loss/FBt2/detection_loss‚3.082198652354154 /loss/FBt3/detection_loss‚2.9546498507261276 % _timestamp‚1.7883988083906662e+09 2loss/SRqqt2/assignment_loss‚1.7130295397979873 _step‚348 0loss/SRqqt3/detection_loss‚1.945471151750915 .loss/FBt4/detection_loss‚0.550741172327268 2loss/SRqqt1/assignment_loss‚1.8059961549303634 1loss/FBt4/assignment_loss‚0.26494391991600535 /loss/FRt4/detection_loss‚1.8729518141065324 1loss/SRqqt4/detection_loss‚1.2625181338301412 epoch‚3 trainer/global_step‚4249 _runtime‚3433.027429065 /loss/FBt2/assignment_loss‚0.606113901186704 /loss/FRt3/detection_loss‚2.6581111330735054 /loss/FRt1/assignment_loss‚3.591949370172289 /loss/FRt2/assignment_loss‚3.704803573457818 0loss/FRt4/assignment_loss‚1.3975490806054096 .loss/FRt1/detection_loss‚2.195348671740956 %loss/total_loss‚46.52201164196674 /loss/FRt2/detection_loss‚2.6540334648207615 1loss/SRqqt1/detection_loss‚1.8824530034623248 2loss/SRqqt3/assignment_loss‚1.7611302169305938XL¹6^ÓØ _step‚348 0loss/SRqqt3/detection_loss‚1.945471151750915 .loss/FBt4/detection_loss‚0.550741172327268 2loss/SRqqt1/assignment_loss‚1.8059961549303634 1loss/FBt4/assignment_loss‚0.26494391991600535 /loss/FRt4/detection_loss‚1.8729518141065324 1loss/SRqqt4/detection_loss‚1.2625181338301412 epoch‚3 trainer/global_step‚4249 _runtime‚3433.027429065 /loss/FBt2/assignment_loss‚0.606113901186704 /loss/FRt3/detection_loss‚2.6581111330735054 /loss/FRt1/assignment_loss‚3.591949370172289 /loss/FRt2/assignment_loss‚3.704803573457818 0loss/FRt4/assignment_loss‚1.3975490806054096 .loss/FRt1/detection_loss‚2.195348671740956 %loss/total_loss‚46.52201164196674 /loss/FRt2/detection_loss‚2.6540334648207615 1loss/SRqqt1/detection_loss‚1.8824530034623248 2loss/SRqqt3/assignment_loss‚1.7611302169305938 /loss/FBt1/detection_loss‚2.2926499359309673 0loss/FRt3/assignment_loss‚4.1166813232396775 2loss/SRqqt4/assignment_loss‚0.8939145084953418 0loss/FBt1/assignment_loss‚0.5824865751783364 0loss/FBt3/assignment_loss‚0.5091993389651179 1loss/SRqqt2/detection_loss‚2.2230871549674442 .loss/FBt2/detection_loss‚3.082198652354154 /loss/FBt3/detection_loss‚2.9546498507261276 % _timestamp‚1.7883988083906662e+09 2loss/SRqqt2/assignment_loss‚1.7130295397979873ܚʈ-‘Ô‚(:† Ú™ãÔÜÉ™® gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚130.775' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.155 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0që ~Õ‚(:ó Ú™ãÔù…µ° network.sent‚ 9.640963e+06) disk./.usagePercent‚15.04815435157101 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0$ memory_percent‚92.47477054595947 disk.rbd1.in‚0% disk./.usageGB‚224.22837448120117 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0$ disk.nvme0n1.out‚20779.435546875" proc.memory.rssMB‚ 2604.2421875 cpu‚25.339528731490503 disk.rbd0.in‚0! disk.dm-0.out‚20779.435546875 disk.sda1.out‚0 disk.nvme0n1.in‚ 338.32421875 proc.cpu.threads‚17 disk.dm-1.in‚ 0.05078125 disk.dm-0.in‚ 338.32421875 disk.sda3.in‚ 0.05078125 disk.sda.out‚ 7541.6953125 disk.sda2.in‚0" disk.nvme0c0n1.in‚ 338.32421875) proc.memory.availableMB‚ 1232.93359375 network.recv‚ 1.225948e+06 disk.dm-1.out‚ 7541.6953125 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 7541.6953125& disk.nvme0c0n1.out‚20779.435546875) proc.memory.percent‚15.89503288269043 disk.sda.in‚ 0.05078125 disk.rbd1.out‚11.98974609375üž>Ö‚(:’ é™ãÔº£® gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚132.956' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.591199999999997 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0wœ`|ׂ(:ñ é™ãÔ’ø²² network.recv‚ 1.231757e+06 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0" disk.nvme0c0n1.in‚ 338.32421875 disk.drbd2266.out‚0% disk.nvme0n1.out‚20779.9013671875% proc.memory.availableMB‚ 1232.9375) proc.memory.percent‚15.89503288269043 disk.sda2.out‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 338.32421875' disk.nvme0c0n1.out‚20779.9013671875 disk.nvme0n1.in‚ 338.32421875" proc.memory.rssMB‚ 2604.2421875 proc.cpu.threads‚17 disk.sda3.out‚ 7542.27734375 disk.rbd1.in‚0) disk./.usagePercent‚15.04815435157101% disk./.usageGB‚224.22837448120117 disk.drbd2265.in‚0 disk.sda1.out‚0 network.sent‚ 9.749436e+06" disk.dm-0.out‚20779.9013671875$ memory_percent‚92.47474670410156 disk.drbd2265.out‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 7542.27734375 disk.sda1.in‚0 disk.rbd1.out‚ 12.052734375 disk.sda.out‚ 7542.27734375 disk.sda2.in‚0 disk.dm-1.in‚ 0.05078125 disk.sda.in‚ 0.05078125 cpu‚25.3383208219204 disk.sda3.in‚ 0.05078125†6ºØ‚(:’ ø™ãÔóâ¡® gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚132.956' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.591199999999997 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽKrñÙ‚(:ö ø™ãÔÅѽ° proc.cpu.threads‚17 disk.sda.out‚ 7542.7265625 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0$ memory_percent‚92.47486591339111) proc.memory.availableMB‚ 1232.91796875 disk.sda3.out‚ 7542.7265625& disk.nvme0n1.out‚20780.41748046875 network.sent‚ 9.755544e+06 disk.nvme0n1.in‚ 338.32421875" proc.memory.rssMB‚ 2604.2421875 disk.sda1.out‚0 disk.sda3.in‚ 0.05078125 disk.rbd1.in‚0 disk.rbd1.out‚ 12.052734375% disk./.usageGB‚224.22837448120117 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 network.recv‚ 1.235722e+06 disk.dm-1.out‚ 7542.7265625 disk.drbd2265.out‚0 cpu‚25.33711067077572 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚20780.41748046875 disk.dm-1.in‚ 0.05078125 disk.drbd2080.out‚0 disk.rbd0.in‚0# disk.dm-0.out‚20780.41748046875) disk./.usagePercent‚15.04815435157101 disk.sda.in‚ 0.05078125 disk.dm-0.in‚ 338.32421875" disk.nvme0c0n1.in‚ 338.32421875) proc.memory.percent‚15.89503288269043†>2Ú- _wandbruntime‚3468  _runtime‚3468L\ñŒÆÛÀ & lr-AdamW/pg2‚0.0001749708554661513 trainer/global_step‚4299 $ _timestamp‚1.788398843463456e+09 _runtime‚3468.100314501 _step‚349 & lr-AdamW/pg1‚0.0001749708554661513ÙW{[ËÜÅ & lr-AdamW/pg1‚0.0001749708554661513 & lr-AdamW/pg2‚0.0001749708554661513 trainer/global_step‚4299 $ _timestamp‚1.788398843463456e+09 _runtime‚3468.100314501 _step‚349Ý}ÿk2Ý- _wandbruntime‚3468  _runtime‚3468þ¼xgÞÕO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚120836433683456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 143256369 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6057205630464 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚120988528415190 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24372049 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57329311 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6057036630528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143236329 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56467033 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6162490925056 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚126893657316864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56467035 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚4299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32957806 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86789334 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86769296 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6162707801088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚126893470313984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚120836433683456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _runtime‚3468.101383297 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚120836451686400 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86769296 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57329855 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24371507 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚350 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6039590802902 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 % _timestamp‚1.7883988434641767e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128041964295168 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248Æ·‚à'ßÚO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 143236329 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚56467033 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6162490925056 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚126893657316864 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚56467035 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚4299 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚32957806 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚86789334 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚86769296 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6162707801088 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚126893470313984 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚120836433683456 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _runtime‚3468.101383297 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚120836451686400 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚86769296 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57329855 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24371507 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚350 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6039590802902 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 % _timestamp‚1.7883988434641767e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚128041964295168 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚120836433683456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 143256369 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6057205630464 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚120988528415190 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24372049 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57329311 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6057036630528 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4299 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32965183 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6041145089058 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6164123686912 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86789083 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚114974897833984 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134234772710400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚121015860762070 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32965181 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86789083 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128070853303296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86809147 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚501 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6058429768704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24377039 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6058598781952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128070877243392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚499 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143288841 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚120863703048192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚120863721051136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚120863721051136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6058598781952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86809147 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143288841 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚351 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57342220 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 143268775 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6040979967446 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6058429768704 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56479694 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56479692 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚126922319833088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24377538 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408œëúQsâãM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚32965183 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6041145089058 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6164123686912 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚86789083 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚114974897833984 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚134234772710400 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚121015860762070 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚32965181 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚86789083 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚128070853303296 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚86809147 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚501 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6058429768704 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24377039 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6058598781952 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚128070877243392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚499 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 143288841 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚120863703048192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚120863721051136 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚120863721051136 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6058598781952 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch“ù:úv_end/active_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚86809147 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 143288841 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚351 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57342220 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 143268775 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6040979967446 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6058429768704 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚56479694 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚56479692 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚126922319833088 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24377538 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6163919407104 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 143268775 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4299ßMG2ã- _wandbruntime‚3468  _runtime‚3468’ 07[äÕ .loss/FRt1/detection_loss‚2.010136297770909 .loss/FRt2/detection_loss‚3.265120166319388 .loss/FRt3/detection_loss‚2.366837136745453 2loss/SRqqt4/assignment_loss‚1.0640140514237335 /loss/FRt4/detection_loss‚1.8782086460029377 /loss/FBt1/detection_loss‚2.2739793929186733 .loss/FBt3/detection_loss‚2.813487857580185 trainer/global_step‚4299 _runtime‚3468.777099193 /loss/FRt2/assignment_loss‚4.087547907122859 0loss/FRt4/assignment_loss‚1.6939631829074784 2loss/SRqqt1/assignment_loss‚1.6223368853430398 0loss/SRqqt2/detection_loss‚2.218350856593161 1loss/SRqqt3/detection_loss‚1.7422080529185962 1loss/FBt4/assignment_loss‚0.07287316050128449 1loss/SRqqt1/detection_loss‚1.8687154145113059 /loss/FBt2/detection_loss‚3.5452734331289926 0loss/FBt3/assignment_loss‚0.8652811497449875 epoch‚3 /loss/FRt3/assignment_loss‚3.084143179655075 2loss/SRqqt2/assignment_loss‚2.2870627614584835 1loss/SRqqt4/detection_loss‚1.0812486991824874 _step‚352 0loss/FRt1/assignment_loss‚3.0242815784045627 2loss/SRqqt3/assignment_loss‚1.5707716740808397 0loss/FBt1/assignment_loss‚0.4715820959721565 1loss/FBt2/assignment_loss‚0.26265251365930453 /loss/FBt4/detection_loss‚0.8168128831411118 &loss/total_loss‚45.986888977087006 % _timestamp‚1.7883988441403637e+09Á1™Y`åÚ 0loss/FRt4/assignment_loss‚1.6939631829074784 2loss/SRqqt1/assignment_loss‚1.6223368853430398 0loss/SRqqt2/detection_loss‚2.218350856593161 1loss/SRqqt3/detection_loss‚1.7422080529185962 1loss/FBt4/assignment_loss‚0.07287316050128449 1loss/SRqqt1/detection_loss‚1.8687154145113059 /loss/FBt2/detection_loss‚3.5452734331289926 0loss/FBt3/assignment_loss‚0.8652811497449875 epoch‚3 /loss/FRt3/assignment_loss‚3.084143179655075 2loss/SRqqt2/assignment_loss‚2.2870627614584835 1loss/SRqqt4/detection_loss‚1.0812486991824874 _step‚352 0loss/FRt1/assignment_loss‚3.0242815784045627 2loss/SRqqt3/assignment_loss‚1.5707716740808397 0loss/FBt1/assignment_loss‚0.4715820959721565 1loss/FBt2/assignment_loss‚0.26265251365930453 /loss/FBt4/detection_loss‚0.8168128831411118 &loss/total_loss‚45.986888977087006 % _timestamp‚1.7883988441403637e+09 .loss/FRt1/detection_loss‚2.010136297770909 .loss/FRt2/detection_loss‚3.265120166319388 .loss/FRt3/detection_loss‚2.366837136745453 2loss/SRqqt4/assignment_loss‚1.0640140514237335 /loss/FRt4/detection_loss‚1.8782086460029377 /loss/FBt1/detection_loss‚2.2739793929186733 .loss/FBt3/detection_loss‚2.813487857580185 trainer/global_step‚4299 _runtime‚3468.777099193 /loss/FRt2/assignment_loss‚4.087547907122859àZõŽœæ‚(:‘ ‡šãÔúÐœ¯ gpu.0.gpu‚30.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚145.89' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.177999999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µÞ¶ïzç‚(:ï ‡šãÔŒ°¸± disk.sda.in‚ 0.05078125 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 7544.8828125 disk.drbd2080.out‚0" proc.memory.rssMB‚ 2604.2421875& disk.nvme0c0n1.out‚20790.158203125 disk.nvme0n1.in‚ 338.33203125 disk.drbd2080.in‚0! disk.dm-0.out‚20790.158203125 disk.sda3.out‚ 7544.8828125$ memory_percent‚92.47379302978516 disk.drbd2265.out‚0 disk.rbd0.out‚0& proc.memory.availableMB‚ 1233.09375 disk.rbd1.out‚12.11572265625 disk.sda.out‚ 7544.8828125 disk.drbd1716.out‚0 disk.dm-0.in‚ 338.33203125 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 9.761943e+06 network.recv‚ 1.240481e+06$ disk./.usageGB‚224.2297821044922 disk.dm-1.in‚ 0.05078125 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.sda1.in‚0) proc.memory.percent‚15.89503288269043* disk./.usagePercent‚15.048248818351146$ disk.nvme0n1.out‚20790.158203125 disk.drbd2266.out‚0 cpu‚25.33586731093882" disk.nvme0c0n1.in‚ 338.33203125 disk.sda3.in‚ 0.05078125õ¨Qaè‚(:’ –šãԢ좯 gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚146.131' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.226200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:x9X|é‚(:ñ –šãÔúÓ£± disk.drbd2080.out‚0 disk.sda1.in‚0" proc.memory.rssMB‚ 2604.2421875 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda.in‚ 0.05078125 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0$ memory_percent‚92.47455596923828 disk.sda3.out‚ 7545.66015625" disk.nvme0c0n1.in‚ 338.33203125 disk.nvme0n1.in‚ 338.33203125 disk.dm-1.out‚ 7545.66015625 disk.drbd2266.in‚0 disk.sda3.in‚ 0.05078125 disk.rbd1.in‚0 disk.dm-1.in‚ 0.05078125 network.sent‚ 9.87009e+06& disk.nvme0c0n1.out‚20790.826171875& proc.memory.availableMB‚ 1232.96875) proc.memory.percent‚15.89503288269043$ disk./.usageGB‚224.2328643798828 disk.sda.out‚ 7545.66015625 disk.rbd0.in‚0 cpu‚25.334580806249782 disk.drbd2265.in‚0 network.recv‚ 1.245679e+06 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 338.33203125! disk.dm-0.out‚20790.826171875) disk./.usagePercent‚15.04845567243881 disk.rbd1.out‚12.20166015625$ disk.nvme0n1.out‚20790.826171875 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0²z2ê- _wandbruntime‚3503  _runtime‚3503 µ“Éëà ' lr-AdamW/pg1‚0.00017375591494640346 ' lr-AdamW/pg2‚0.00017375591494640346 trainer/global_step‚4349 % _timestamp‚1.7883988790949433e+09 _runtime‚3503.731789855 _step‚353xVÄÎìÈ ' lr-AdamW/pg1‚0.00017375591494640346 ' lr-AdamW/pg2‚0.00017375591494640346 trainer/global_step‚4349 % _timestamp‚1.7883988790949433e+09 _runtime‚3503.731789855 _step‚353áüs÷`2í- _wandbruntime‚3503  _runtime‚3503 Û'îÕO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144882021 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚122199919923200 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚122355159350398 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87782036 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87761998 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33326656 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚135721196873728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚116246096729088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚135720968640512 _runtime‚3503.732668412 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚122199901920256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚128326609873408 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6126876953088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33326654 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24641989 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚128326796876288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87782036 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204293120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6233380874240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚122355341511162 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87761998 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚122199901920256 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚116246113768448 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6109227742714 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57968645 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚122199919923200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚354 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚128326796876288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57099985 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57099983 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57968118 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 % _timestamp‚1.7883988790956461e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144861981 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57099983 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128}æJ*à'ïÚO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚122199901920256 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚128326609873408 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6126876953088 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚33326654 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚24641989 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚128326796876288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚87782036 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204293120 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6233380874240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚122355341511162 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚87761998 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚122199901920256 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚116246113768448 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6109227742714 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚57968645 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚122199919923200 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚354 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚128326796876288 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57099985 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57099983 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚57968118 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 % _timestamp‚1.7883988790956461e+09 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 144861981 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚57099983 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 144882021 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚122199919923200 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚122355159350398 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚87782036 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚87761998 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33326656 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚135721196873728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚116246096729088 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚135720968640512 _runtime‚3503.732668412 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160â3h$ù2ð- _wandbruntime‚3504  _runtime‚3504è¶lñßM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144894427 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚135751292962304 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6234792247808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6128270104576 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6128101091328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚122227171284992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚122227171284992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6235011207680 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 NADeviceStatsMonitor.on_train_batch_end/allocation.small_p9e{£Ïool.freed‚87781785 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚122382491697278 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚518 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚128355272376320 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144914493 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57112644 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87781785 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24647452 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6110451785854 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚128355459392512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚3504.404646229 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚128355459392512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57112642 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57980967 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚135751535862272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6110616907466 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57112642 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6128270104576 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚129516500714496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33334031 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚116272056950784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚122227189287936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚129516524654592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚516 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _step‚355 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144914493 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57112644 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24646936 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0†~ê&òäM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚122227189287936 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚129516524654592 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚516 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _step‚355 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 144914493 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57112644 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24646936 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 144894427 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚135751292962304 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6234792247808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6128270104576 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6128101091328 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚122227171284992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚122227171284992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6235011207680 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚87781785 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚122382491697278 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚518 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚128355272376320 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 144914493 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚57112644 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚87781785 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24647452 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6110451785854 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚128355459392512 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _runtime‚3504.404646229 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚128355459392512 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57112642 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚57980967 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚135751535862272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6110616907466 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚57112642 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6128270104576 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚129516500714496 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33334031 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚116272056950784 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192ãG½³2ó- _wandbruntime‚3504  _runtime‚3504yàËZôÔ _step‚356 0loss/FBt2/assignment_loss‚0.3812098171260345 1loss/SRqqt2/detection_loss‚2.7334194892439347 /loss/FBt1/detection_loss‚3.4288775846362114 0loss/FBt3/assignment_loss‚0.6196298014372588 /loss/FRt1/detection_loss‚2.4995762306180866 epoch‚3 % _timestamp‚1.7883988797689729e+09 0loss/FBt1/assignment_loss‚0.3591466708016924 /loss/FRt3/detection_loss‚2.3318711417642506 0loss/SRqqt1/assignment_loss‚2.44272163271904 /loss/FRt2/detection_loss‚3.3068081706762316 1loss/SRqqt3/detection_loss‚1.6403482963259404 2loss/SRqqt2/assignment_loss‚2.0716081713807992 1loss/SRqqt1/detection_loss‚2.0233556112647055 /loss/FBt4/detection_loss‚0.8207592090429592 %loss/total_loss‚48.13435467134063 trainer/global_step‚4349 _runtime‚3504.405704174 /loss/FRt1/assignment_loss‚3.132510533387011 1loss/SRqqt4/detection_loss‚1.1657649793085598 /loss/FRt2/assignment_loss‚4.290564751625061 /loss/FRt3/assignment_loss‚3.215412134473974 0loss/FRt4/assignment_loss‚1.6480453541645637 2loss/SRqqt3/assignment_loss‚1.2739552940743475 2loss/SRqqt4/assignment_loss‚0.7905406480625509 1loss/FBt4/assignment_loss‚0.18759235788704803 /loss/FRt4/detection_loss‚1.8086786392646341 .loss/FBt2/detection_loss‚3.323387920856476 /loss/FBt3/detection_loss‚2.6385702311992647Ð%œ_õÙ 0loss/FBt3/assignment_loss‚0.6196298014372588 /loss/FRt1/detection_loss‚2.4995762306180866 epoch‚3 % _timestamp‚1.7883988797689729e+09 0loss/FBt1/assignment_loss‚0.3591466708016924 /loss/FRt3/detection_loss‚2.3318711417642506 0loss/SRqqt1/assignment_loss‚2.44272163271904 /loss/FRt2/detection_loss‚3.3068081706762316 1loss/SRqqt3/detection_loss‚1.6403482963259404 2loss/SRqqt2/assignment_loss‚2.0716081713807992 1loss/SRqqt1/detection_loss‚2.0233556112647055 /loss/FBt4/detection_loss‚0.8207592090429592 %loss/total_loss‚48.13435467134063 trainer/global_step‚4349 _runtime‚3504.405704174 /loss/FRt1/assignment_loss‚3.132510533387011 1loss/SRqqt4/detection_loss‚1.1657649793085598 /loss/FRt2/assignment_loss‚4.290564751625061 /loss/FRt3/assignment_loss‚3.215412134473974 0loss/FRt4/assignment_loss‚1.6480453541645637 2loss/SRqqt3/assignment_loss‚1.2739552940743475 2loss/SRqqt4/assignment_loss‚0.7905406480625509 1loss/FBt4/assignment_loss‚0.18759235788704803 /loss/FRt4/detection_loss‚1.8086786392646341 .loss/FBt2/detection_loss‚3.323387920856476 /loss/FBt3/detection_loss‚2.6385702311992647 _step‚356 0loss/FBt2/assignment_loss‚0.3812098171260345 1loss/SRqqt2/detection_loss‚2.7334194892439347 /loss/FBt1/detection_loss‚3.4288775846362114ä<îÇ0’ö‚(:‡ ¥šãÔíÞ»® gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚144.287' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.8574 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®ˆ¹Ï…÷‚(:ú ¥šãÔ뻚² disk.drbd2265.out‚0 disk.sda3.out‚ 7546.0546875" proc.memory.rssMB‚ 2604.2421875 disk.drbd2266.in‚0 disk.sda3.in‚ 0.05078125 disk.sda2.out‚0 disk.dm-1.in‚ 0.05078125 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 7546.0546875 proc.cpu.threads‚17% disk./.usageGB‚224.23247528076172 disk.rbd0.in‚0 disk.dm-0.in‚ 338.33203125 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 338.33203125 disk.sda.out‚ 7546.0546875$ memory_percent‚92.47567653656006* disk./.usagePercent‚15.048429559670318 disk.sda1.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 cpu‚25.333430060767764 network.sent‚ 9.875037e+06 disk.drbd2266.out‚0 disk.drbd2080.in‚0# disk.dm-0.out‚20798.64111328125 disk.rbd1.out‚12.26416015625 disk.sda.in‚ 0.05078125 network.recv‚ 1.249173e+06 disk.drbd2265.in‚0( disk.nvme0c0n1.out‚20798.64111328125 disk.nvme0n1.in‚ 338.33203125& disk.nvme0n1.out‚20798.64111328125 disk.drbd1716.in‚0) proc.memory.availableMB‚ 1232.78515625) proc.memory.percent‚15.89503288269043ðï ø‚(:’ ´šãÔÄç›® gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚132.419' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.483800000000002 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“1N©€ù‚(:õ ´šãÔÂ霰# disk.dm-0.out‚20798.81689453125& disk.nvme0n1.out‚20798.81689453125 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.sda3.out‚ 7546.828125 disk.rbd1.out‚12.26416015625# memory_percent‚92.4757719039917 disk.sda.out‚ 7546.828125 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0" proc.memory.rssMB‚ 2604.2421875) proc.memory.percent‚15.89503288269043 cpu‚25.33219128451613 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.dm-0.in‚ 338.33203125 disk.sda1.in‚0 disk.sda3.in‚ 0.05078125 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 338.33203125 network.recv‚ 1.255325e+06 disk.sda.in‚ 0.05078125( disk.nvme0c0n1.out‚20798.81689453125) proc.memory.availableMB‚ 1232.76953125 proc.cpu.threads‚17 disk.nvme0n1.in‚ 338.33203125 disk.dm-1.in‚ 0.05078125 disk.dm-1.out‚ 7546.828125 disk.drbd2266.out‚0 network.sent‚ 9.983598e+06* disk./.usagePercent‚15.048429815677853% disk./.usageGB‚224.23247909545898 disk.rbd0.in‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0zöÑ2ú- _wandbruntime‚3539  _runtime‚3539åÊ2pÈû ' lr-AdamW/pg1‚0.00017252716539843888 ' lr-AdamW/pg2‚0.00017252716539843888 trainer/global_step‚4399 % _timestamp‚1.7883989147211642e+09 _runtime‚ 3539.35802018 _step‚357QcÞ“ÍüÇ _step‚357 ' lr-AdamW/pg1‚0.00017252716539843888 ' lr-AdamW/pg2‚0.00017252716539843888 trainer/global_step‚4399 % _timestamp‚1.7883989147211642e+09 _runtime‚ 3539.35802018å7»2ý- _wandbruntime‚3539  _runtime‚3539o9ÍþÕO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221070336 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129759749432832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33695506 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚130933235177472 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚123563370157056 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6196379275776 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58608557 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6196379275776 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58608008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130933259117568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6304481729024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129759749432832 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 trainer/global_step‚4399 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6178699561122 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚123563388160000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57732933 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚129759936435712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6196548275712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚137237740846592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245010432 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚117543272885248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117543255845888 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146507673 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 146487633 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57732935 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚137237495836160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚88754700 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚358 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚88774738 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚123721972446370 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6178534439718 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚57732935 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚88774738 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚24912504 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚549 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129759936435712 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚129759936435712 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚129759749432832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚33695506 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚130933235177472 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚123563370157056 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6196379275776 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚58608557 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6196379275776 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚58608008 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚130933259117568 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6304481729024 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚129759749432832 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 trainer/global_step‚4399 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6178699561122 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚123563388160000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚57732933 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚129759936435712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6196548275712 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚137237740846592 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245010432 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚117543272885248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚117543255845888 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 146507673 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 146487633 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚57732935 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040æ' 0‡€‚(:’ ÚãÔÛȸ® gpu.0.gpu‚32.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚126.468' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.293599999999998 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0=—Ö¡q‚(:æ ÚãÔÍ´„± disk.dm-0.in‚ 338.359375 disk.sda3.in‚ 0.05078125 disk.rbd1.out‚12.29638671875 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 338.359375 disk.sda2.out‚0 network.recv‚ 1.26006e+06 disk.dm-1.out‚ 7547.2734375 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 338.359375 disk.sda.out‚ 7547.2734375 disk.sda2.in‚0 cpu‚25.33099190928755$ disk./.usageGB‚225.2366180419922 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 7547.2734375 disk.drbd2265.out‚0 disk.dm-0.out‚20799.25390625 disk.sda1.out‚0$ memory_percent‚92.47260093688965( proc.memory.availableMB‚ 1233.2890625) proc.memory.percent‚15.89503288269043 network.sent‚ 9.989945e+06 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.05078125 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.sda.in‚ 0.05078125" proc.memory.rssMB‚ 2604.2421875% disk.nvme0c0n1.out‚20799.25390625# disk.nvme0n1.out‚20799.25390625) disk./.usagePercent‚15.11581842290839 disk.drbd1716.in‚0>òÿ82‚- _wandbruntime‚3540  _runtime‚3540bé+šå&ƒßM >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6179923604262 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚123590657524736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117569216067584 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146520079 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24918362 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚137267826311168 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58621245 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6305888756736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 % _timestamp‚1.7883989154013002e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6197941427200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88774487 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚123590657524736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6197941427200 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3540.038524908 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117569199028224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6305678185472 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146540145 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚130962148125696 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚129788598951936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58620703 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33702881 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24917822 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚123590639521792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88794551 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4399 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚123749122632486 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57745594 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57745592’üÑ,¼"„äM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚57745594 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚57745592 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6179923604262 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚123590657524736 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚117569216067584 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 146520079 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚24918362 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚137267826311168 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚58621245 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6305888756736 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 % _timestamp‚1.7883989154013002e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6197941427200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚88774487 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚123590657524736 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6197941427200 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3540.038524908 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚117569199028224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6305678185472 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 146540145 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚130962148125696 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚129788598951936 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚58620703 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚33702881 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚24917822 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚123590639521792 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚88794551 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 trainer/global_step‚4399 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_' Û©.pool.peak‚26 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚123749122632486 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0çÅ3•"2…- _wandbruntime‚3540  _runtime‚3540ù.˜ R†Ì 1loss/SRqqt1/assignment_loss‚1.496523350097087 .loss/FBt4/detection_loss‚0.767015720076606 &loss/total_loss‚47.243515254857954 _runtime‚3540.039945317 0loss/FRt1/assignment_loss‚3.3579627441578226 1loss/SRqqt3/detection_loss‚1.9189935165877436 epoch‚3 /loss/FRt2/assignment_loss‚4.025976639527541 .loss/FRt1/detection_loss‚2.160478295849972 .loss/FRt3/assignment_loss‚2.94364241484938 2loss/SRqqt3/assignment_loss‚1.6184180574309008 0loss/FBt2/assignment_loss‚0.5004012228907243 0loss/FBt3/assignment_loss‚0.6340311982348794 0loss/FBt4/assignment_loss‚0.3052004362475698 _step‚360 0loss/FBt1/assignment_loss‚0.7608436986791502 1loss/SRqqt1/detection_loss‚1.8293753816765181 0loss/SRqqt2/detection_loss‚2.616631457099208 /loss/FBt3/detection_loss‚3.0345501651366553 trainer/global_step‚4399 2loss/SRqqt4/assignment_loss‚1.0925410891631766 .loss/FRt2/detection_loss‚2.917963628585522 /loss/FRt4/detection_loss‚1.7631059868032473 0loss/SRqqt4/detection_loss‚1.330800266847724 /loss/FBt1/detection_loss‚2.2509894549846647 0loss/SRqqt2/assignment_loss‚2.12275117525348 /loss/FRt3/detection_loss‚3.0028023041527847 ,loss/FBt2/detection_loss‚3.0497030377388 % _timestamp‚1.7883989154031777e+09 0loss/FRt4/assignment_loss‚1.7428140127867984…+,W‡Ñ 1loss/SRqqt1/detection_loss‚1.8293753816765181 0loss/SRqqt2/detection_loss‚2.616631457099208 /loss/FBt3/detection_loss‚3.0345501651366553 trainer/global_step‚4399 2loss/SRqqt4/assignment_loss‚1.0925410891631766 .loss/FRt2/detection_loss‚2.917963628585522 /loss/FRt4/detection_loss‚1.7631059868032473 0loss/SRqqt4/detection_loss‚1.330800266847724 /loss/FBt1/detection_loss‚2.2509894549846647 0loss/SRqqt2/assignment_loss‚2.12275117525348 /loss/FRt3/detection_loss‚3.0028023041527847 ,loss/FBt2/detection_loss‚3.0497030377388 % _timestamp‚1.7883989154031777e+09 0loss/FRt4/assignment_loss‚1.7428140127867984 1loss/SRqqt1/assignment_loss‚1.496523350097087 .loss/FBt4/detection_loss‚0.767015720076606 &loss/total_loss‚47.243515254857954 _runtime‚3540.039945317 0loss/FRt1/assignment_loss‚3.3579627441578226 1loss/SRqqt3/detection_loss‚1.9189935165877436 epoch‚3 /loss/FRt2/assignment_loss‚4.025976639527541 .loss/FRt1/detection_loss‚2.160478295849972 .loss/FRt3/assignment_loss‚2.94364241484938 2loss/SRqqt3/assignment_loss‚1.6184180574309008 0loss/FBt2/assignment_loss‚0.5004012228907243 0loss/FBt3/assignment_loss‚0.6340311982348794 0loss/FBt4/assignment_loss‚0.3052004362475698 _step‚360 0loss/FBt1/assignment_loss‚0.7608436986791502èƒBˆ‚(:’ ÒšãÔ«Û® gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚111.112' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.222399999999997 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—–…Ùx‰‚(:í ÒšãÔúÓ¹°" disk.dm-0.out‚22157.2802734375 disk.sda1.out‚0 disk.rbd1.out‚12.29638671875 network.sent‚ 1.0086574e+07 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.dm-0.in‚ 341.40625 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 341.40625 disk.rbd0.in‚0 disk.sda3.in‚ 0.05078125' disk.nvme0c0n1.out‚22157.2802734375% disk.nvme0n1.out‚22157.2802734375" proc.memory.rssMB‚ 2604.2421875 disk.drbd2080.out‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 1233.203125 cpu‚25.329753366753568) proc.memory.percent‚15.89503288269043 disk.sda.in‚ 0.05078125 disk.sda.out‚ 7547.8046875 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.sda3.out‚ 7547.8046875$ memory_percent‚92.47312545776367* disk./.usagePercent‚15.137453107611968 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 341.40625 disk.sda2.out‚0 disk.dm-1.in‚ 0.05078125 disk.dm-1.out‚ 7547.8046875 disk.drbd2266.in‚0 proc.cpu.threads‚17 network.recv‚ 1.265353e+06% disk./.usageGB‚225.55899047851562 disk.rbd0.out‚0ëùÿ‘Š‚(:† ášãÔ«”ð­ gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚139.685' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.937 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µöNj‹‚(:ß ášãÔ¿®é° disk.sda3.out‚ 7548.48046875 disk.nvme0c0n1.in‚342.5 disk.sda.out‚ 7548.48046875 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 network.recv‚ 1.271094e+06 disk.rbd1.in‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.05078125 disk.drbd2266.out‚0 network.sent‚ 1.0184344e+07$ memory_percent‚92.47617721557617 disk.dm-0.in‚342.5" disk.dm-0.out‚22159.5908203125 disk.sda1.out‚0' disk.nvme0c0n1.out‚22159.5908203125" proc.memory.rssMB‚ 2604.2421875 disk.rbd0.out‚0 disk.sda.in‚ 0.05078125 disk.drbd1716.in‚0 cpu‚25.328511289892933 disk.sda1.in‚0 disk.sda3.in‚ 0.05078125 disk.drbd2266.in‚0 disk.drbd2265.out‚0' proc.memory.availableMB‚ 1232.703125* disk./.usagePercent‚15.137482036463338% disk./.usageGB‚225.55942153930664 disk.rbd1.out‚ 12.359375 disk.drbd2080.out‚0) proc.memory.percent‚15.89503288269043 proc.cpu.threads‚17 disk.nvme0n1.in‚342.5% disk.nvme0n1.out‚22159.5908203125 disk.drbd1716.out‚0 disk.dm-1.out‚ 7548.48046875 disk.rbd0.in‚0x„ù2Œ- _wandbruntime‚3575  _runtime‚3575€Ô^sÉà ' lr-AdamW/pg1‚0.00017128486289182164 ' lr-AdamW/pg2‚0.00017128486289182164 trainer/global_step‚4449 % _timestamp‚1.7883989505493298e+09 _runtime‚3575.186292891 _step‚361ZkO/ÎŽÈ _runtime‚3575.186292891 _step‚361 ' lr-AdamW/pg1‚0.00017128486289182164 ' lr-AdamW/pg2‚0.00017128486289182164 trainer/global_step‚4449 % _timestamp‚1.7883989505493298e+09é ó`M2- _wandbruntime‚3575  _runtime‚3575Ûùd Ù'ÓO L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚89747402 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 148113285 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34064354 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 # _timestamp‚1.78839895055004e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚131193075995136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚118840414962688 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚58365885 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6266050598400 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚89767440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚132378906528768 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚131193075995136 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6248171379530 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚58365885 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25181551 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25181024 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚89767440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚132378882588672 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59245907 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58365883 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚124926856396800 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148133325 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6375307545600 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6266219598336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89747402 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6248006258126 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚362 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131192888992256 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59245907 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚58365883 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚124926856396800 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 148133325 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6375307545600 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6266219598336 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚89747402 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6248006258126 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚362 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚131192888992256 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4449 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89767189 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148165797 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚125115753567694 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚124954107758592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58378542 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59258066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25186335 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚138784542701056 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6376508458496 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59258593 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ óÏR406789969 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58378542 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34071733 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6249560544282 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131221551495168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 _runtime‚3575.858989228 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚138784303995392 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚124954107758592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6267612749824 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6267443736576 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58378544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 148145731 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148165797 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89787253 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89787253 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89767189 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚125115935728666 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0À#—ê&”äM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚124954125761536 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚131221738511360 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34071731 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚58378542 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59258066 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25186335 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚138784542701056 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6376508458496 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59258593 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚58378542 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34071733 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6249560544282 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚131221551495168 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 _runtime‚3575.858989228 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚138784303995392 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚124954107758592 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6267612749824 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6267443736576 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚58378544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 148145731 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 148165797 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚89787253 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚89787253 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚89767189 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚125115935728666 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚363 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚132407795536896 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6267443736576 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚58378544 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 % _timestamp‚1.7883989512215347e+09 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6267612749824 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚118866358145024 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚118866375184384 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6249395422670 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚124954125761536 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚4449 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚89767189 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 148165797 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚125115753567694 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚124954107758592 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 EDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59887530 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚132626028551680 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6335721921024 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149738937 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34433204 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6446192178176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6317643197938 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚132626028551680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133824553939968 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚120137591118848 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90760142 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚126290324633600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788398986433799e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6335721921024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90740104 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚140270746118144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90760142 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚132626215554560 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚126290306630656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25454324 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚4499 _step‚366 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6317478076534 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6445979496448 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚120137574079488 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚132626215554560 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚59887530 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_stO;Š°@$art/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚132626028551680 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6335721921024 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 149738937 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚34433204 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6446192178176 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6317643197938 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚132626028551680 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚133824553939968 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚120137591118848 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚90760142 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚126290324633600 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788398986433799e+09 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6335721921024 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚90740104 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚140270746118144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚90760142 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚132626215554560 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚126290306630656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25454324 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚4499 _step‚366 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6317478076534 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6445979496448 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚120137574079488 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚132626215554560 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90779955 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 trainer/global_step‚4499 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6318867241078 _runtime‚3611.742561392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚520 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6319032362690 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚126317575995392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59900212 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59011494 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6337284072448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140301076383232 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149791449 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚132654878070784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚120163534301184 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90759891 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7883989871050646e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59899692 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚126482566663874 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚120163517261824 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6337115059200 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90779955 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140300841871872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6337284072448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149791449 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59011492 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6337115059200 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _step‚367 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59011494m¹« ê&¤äM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59011492 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚34440583 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25459629 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 149791449 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚132654878070784 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚120163534301184 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚90759891 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7883989871050646e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚59899692 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚126482566663874 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚120163517261824 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6337115059200 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚90779955 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚140300841871872 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6337284072448 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 149791449 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59011492 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6337115059200 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _step‚367 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59011494 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34440581 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚90779955 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 trainer/global_step‚4499 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6318867241078 _runtime‚3611.742561392 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚520 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6319032362690 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚126317575995392 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚59900212 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59011494 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6337284072448 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚140301076383232 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 PDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221070336 _step‚370 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚121434750235648 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59631783 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60524320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚134059168111104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚134059355113984 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60523811 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135270201351168 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6517025974272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚4549 % _timestamp‚1.7883990221340475e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚127653792870400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚121434733196288 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚134059355113984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151384629 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6386949894942 ‹Ôà'³ÚO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚34802056 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚59631783 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚91732806 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚141787227325440 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚134059355113984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 151384629 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6386949894942 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚91752844 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 151364589 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25721757 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚127821865251994 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221070336 _step‚370 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚121434750235648 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚59631783 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚60524320 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚134059168111104 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚134059355113984 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚60523811 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚135270201351168 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6517025974272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚4549 % _timestamp‚1.7883990221340475e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚127653792870400 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚121434733196288 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6406955395072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6406786381824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202182656 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4549 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚121460693417984 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91752593 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚134088017630208 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6406786381824 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226122752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135299090359296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚127681062235136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚127681044232192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚134087830614016 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚127681062235136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6518231424512 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6388339059486 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59644442 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91752593 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59644444 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34809431 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151397035 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59644442 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60536369 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59644444 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚121460676378624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141817547906560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91772657 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151417101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25727455 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25726938 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚141817321783808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127849015438110 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 $ _timestamp‚1.788399023035045e+09 _step‚371 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151397035 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127849197599082UúÔxé&¶ãM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚134087830614016 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚127681062235136 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6518231424512 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6388339059486 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚59644442 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚91752593 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚59644444 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚34809431 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 151397035 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚59644442 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚60536369 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚59644444 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚121460676378624 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚141817547906560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚91772657 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 151417101 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25727455 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25726938 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚141817321783808 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚127849015438110 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 $ _timestamp‚1.788399023035045e+09 _step‚371 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 151397035 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚127849197599082 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6406955395072 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6406786381824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202182656 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚4549 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚121460693417984 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚91752593 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚134088017630208 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6406786381824 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226122752 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚135299090359296 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚127681062235136 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚127681044232192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2óM)ûÑ2·- _wandbruntime‚3647  _runtime‚3647Zx(V¸Ð epoch‚3 /loss/FRt3/assignment_loss‚3.416708379257016 .loss/FRt1/detection_loss‚2.165956936776638 0loss/SRqqt2/detection_loss‚2.794331363269261 /loss/FBt4/detection_loss‚0.8398057226848323 _step‚372 2loss/SRqqt1/assignment_loss‚1.5309822767972947 /loss/FRt2/detection_loss‚3.2754264059274094 /loss/FBt3/detection_loss‚2.2873449524243674 %loss/total_loss‚47.51665611596508 0loss/FBt2/assignment_loss‚0.1763244473074792 1loss/SRqqt4/detection_loss‚1.0543436600128189 trainer/global_step‚4549 _runtime‚3647.673955168 /loss/FRt2/assignment_loss‚4.259035411088363 2loss/SRqqt3/assignment_loss‚1.4757624721347258 /loss/FRt4/detection_loss‚1.8107067558500503 1loss/SRqqt1/detection_loss‚1.7038326368882106 1loss/SRqqt2/assignment_loss‚2.993972842714616 2loss/SRqqt4/assignment_loss‚1.1768293677014299 1loss/FBt3/assignment_loss‚0.45208517958720523 % _timestamp‚1.7883990230372012e+09 /loss/FRt1/assignment_loss‚3.041486183447497 0loss/FRt4/assignment_loss‚1.6458404011196561 0loss/FBt1/assignment_loss‚0.0978103583403416 .loss/FBt1/detection_loss‚3.100052927221571 .loss/FBt2/detection_loss‚3.846217818558216 1loss/FBt4/assignment_loss‚0.14857968066730143 -loss/FRt3/detection_loss‚2.41269487160735 0loss/SRqqt3/detection_loss‚1.810525064581427ýdxU[¹Õ /loss/FRt2/detection_loss‚3.2754264059274094 /loss/FBt3/detection_loss‚2.2873449524243674 %loss/total_loss‚47.51665611596508 0loss/FBt2/assignment_loss‚0.1763244473074792 1loss/SRqqt4/detection_loss‚1.0543436600128189 trainer/global_step‚4549 _runtime‚3647.673955168 /loss/FRt2/assignment_loss‚4.259035411088363 2loss/SRqqt3/assignment_loss‚1.4757624721347258 /loss/FRt4/detection_loss‚1.8107067558500503 1loss/SRqqt1/detection_loss‚1.7038326368882106 1loss/SRqqt2/assignment_loss‚2.993972842714616 2loss/SRqqt4/assignment_loss‚1.1768293677014299 1loss/FBt3/assignment_loss‚0.45208517958720523 % _timestamp‚1.7883990230372012e+09 /loss/FRt1/assignment_loss‚3.041486183447497 0loss/FRt4/assignment_loss‚1.6458404011196561 0loss/FBt1/assignment_loss‚0.0978103583403416 .loss/FBt1/detection_loss‚3.100052927221571 .loss/FBt2/detection_loss‚3.846217818558216 1loss/FBt4/assignment_loss‚0.14857968066730143 -loss/FRt3/detection_loss‚2.41269487160735 0loss/SRqqt3/detection_loss‚1.810525064581427 epoch‚3 /loss/FRt3/assignment_loss‚3.416708379257016 .loss/FRt1/detection_loss‚2.165956936776638 0loss/SRqqt2/detection_loss‚2.794331363269261 /loss/FBt4/detection_loss‚0.8398057226848323 _step‚372 2loss/SRqqt1/assignment_loss‚1.5309822767972947ôr !&’º‚(:‡ »›ãÔ–£·® gpu.0.gpu‚22.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚135.378' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.0756 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž5¶“~»‚(:ó »›ãÔ‘ý ±! disk.nvme0c0n1.in‚ 342.5078125 disk.drbd2266.out‚0! disk.dm-0.out‚22176.318359375& disk.nvme0c0n1.out‚22176.318359375$ disk.nvme0n1.out‚22176.318359375 disk.dm-1.in‚ 0.05078125% disk./.usageGB‚225.55995178222656 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 7553.78515625 disk.drbd1716.out‚0" proc.memory.rssMB‚ 2604.2421875 network.sent‚ 1.043361e+07* disk./.usagePercent‚15.137517621510597 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 7553.78515625 disk.nvme0n1.in‚ 342.5078125 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 7553.78515625 disk.drbd2266.in‚0# memory_percent‚92.4767255783081 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 342.5078125 disk.sda3.in‚ 0.05078125 disk.sda.in‚ 0.05078125 cpu‚25.321702618974967 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.rbd1.out‚12.42236328125) proc.memory.availableMB‚ 1232.61328125 network.recv‚ 1.315544e+06 disk.sda1.in‚0) proc.memory.percent‚15.89503288269043Sù¢¼‚(:’ Ê›ãÔßÍ¥° gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚147.246' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.449200000000005 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯k3Tx½‚(:í Ê›ãÔÀºÁ² disk.rbd0.out‚0) disk./.usagePercent‚15.13741086636882& disk.nvme0c0n1.out‚22176.568359375 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda.in‚ 0.05078125! disk.dm-0.out‚22176.568359375$ memory_percent‚92.47939586639404) proc.memory.availableMB‚ 1232.17578125 disk.drbd1716.in‚0$ disk.nvme0n1.out‚22176.568359375 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.05078125 disk.rbd1.in‚0 disk.sda.out‚ 7554.234375) proc.memory.percent‚15.89505672454834 proc.cpu.threads‚17 disk.dm-0.in‚ 342.5078125 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 342.5078125 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.rbd1.out‚12.42236328125# proc.memory.rssMB‚ 2604.24609375 cpu‚25.320515794137947 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.sda3.out‚ 7554.234375 network.sent‚ 1.0541496e+07 network.recv‚ 1.32086e+06! disk.nvme0c0n1.in‚ 342.5078125 disk.sda1.in‚0$ disk./.usageGB‚225.5583610534668 disk.dm-1.in‚ 0.05078125 disk.dm-1.out‚ 7554.234375 ?2¾- _wandbruntime‚3682  _runtime‚3682 Þ,Ç¿Á trainer/global_step‚4599 % _timestamp‚1.7883990580825508e+09 _runtime‚3682.719418395 _step‚373 & lr-AdamW/pg1‚0.0001674792403576428 & lr-AdamW/pg2‚0.0001674792403576428ûˆfÔÌÀÆ trainer/global_step‚4599 % _timestamp‚1.7883990580825508e+09 _runtime‚3682.719418395 _step‚373 & lr-AdamW/pg1‚0.0001674792403576428 & lr-AdamW/pg2‚0.0001674792403576428õ¸Jù2Á- _wandbruntime‚3682  _runtime‚3682#”íÛ'ÂÕO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚135492307670528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25990786 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135492307670528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60264735 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚135492494673408 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61161162 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚3682.720342729 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60264735 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92725508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129188314026438 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 trainer/global_step‚4599 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6587706802688 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 % _timestamp‚1.7883990580832658e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129017243104256 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25990258 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129017261107200 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153010281 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚122731892313088 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143303531624960 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚136715848762368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129017261107200 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416ê:Ž?#ÃÚO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60264733 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚122731909352448 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6587917387264 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚135492307670528 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚60264735 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚135492494673408 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚61161162 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _runtime‚3682.720342729 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚60264735 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚92725508 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚129188314026438 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 trainer/global_step‚4599 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6587706802688 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 % _timestamp‚1.7883990580832658e+09 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚129017243104256 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚25990258 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚129017261107200 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 153010281 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚122731892313088 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚143303531624960 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚136715848762368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚129017261107200 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚135492307670528 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚25990786 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153042753 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35178281 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153022687 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6589334032384 _step‚375 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129044512468992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 $ _timestamp‚1.788399058761435e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129044530471936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35178283 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6589117169664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153042753 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61174442 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚136744737770496 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚122757852534784 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92765359 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129044512468992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60277392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25995634 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚136744761710592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 _runtime‚3683.398693634 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚135521157189632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6457810877894 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61173915 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129215828534290 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚4599 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6476457704448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129044530471936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6476626717696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6476457704448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60277394 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6476626717696 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25996159 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143334095742976 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92745295 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6457975999506 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60277392 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143333854940160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚122757835495424 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92745295 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚129215646373318DU?1é&ÆãM VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 153042753 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35178281 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 153022687 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6589334032384 _step‚375 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚129044512468992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 $ _timestamp‚1.788399058761435e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚129044530471936 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35178283 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6589117169664 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 153042753 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61174442 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚136744737770496 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚122757852534784 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚92765359 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚129044512468992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60277392 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚25995634 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚136744761710592 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 _runtime‚3683.398693634 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚135521157189632 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6457810877894 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61173915 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚129215828534290 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚4599 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6476457704448 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚129044530471936 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6476626717696 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6476457704448 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60277394 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6476626717696 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚25996159 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚143334095742976 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚92745295 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6457975999506 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60277392 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚143333854940160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚122757835495424 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚92745295 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚129215646373318 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚136925634232832 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚124029051429888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6526058653162 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6544735888896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26261279 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154615893 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚130380711341056 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60897683 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35539754 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93718210 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154635933 % _timestamp‚1.7883990944908092e+09 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26261799 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚136925447229952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6544735888896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6544904888832 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚138161472233472 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚136925634232832 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚124029051429888 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6526058653162 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6544735888896 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26261279 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 154615893 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚130380711341056 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚60897683 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚35539754 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚93718210 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 154635933 % _timestamp‚1.7883990944908092e+09 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26261799 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚136925447229952 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6544735888896 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6544904888832 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚138161472233472 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3719.809089879 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154668405 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚130407980705792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚130407980705792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚138190409121792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚130407998708736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚144850690525696 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60910342 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154668405 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60910344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26266793 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚144850456014336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6527447817914 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚136954296749056 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93737997 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35547131 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚130582459469498 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93758061 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35547133 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚136954109732864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138190385181696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6527282696302 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61814460 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93758061 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6660281403904 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154648339 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6546129027072 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61813924 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _step‚379 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60910344 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚124055011651584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60910342 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6660070832640 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 trainer/global_step‚4649 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93737997 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217d³åÖäM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚130407998708736 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚144850690525696 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚60910342 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 154668405 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚60910344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26266793 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚144850456014336 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6527447817914 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMt«¯Èonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚136954296749056 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚93737997 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35547131 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚130582459469498 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚93758061 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35547133 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚136954109732864 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚138190385181696 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6527282696302 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚61814460 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚93758061 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6660281403904 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 154648339 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6546129027072 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚61813924 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _step‚379 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚60910344 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚124055011651584 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚60910342 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6660070832640 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 trainer/global_step‚4649 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚93737997 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 % _timestamp‚1.7883990951718788e+09 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6546298040320 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚3719.809089879 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 154668405 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚130407980705792 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚130407980705792 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚138190409121792 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217û6÷g,2×- _wandbruntime‚3719  _runtime‚3719ìMˆ¨\ØÖ /loss/FBt3/detection_loss‚2.4915862500667574 /loss/FRt2/detection_loss‚2.8721609357744455 0loss/FRt2/assignment_loss‚4.3684662990272045 1loss/SRqqt4/assignment_loss‚1.244445913686202 0loss/SRqqt1/detection_loss‚1.773145123251847 1loss/SRqqt4/detection_loss‚1.1480600734169666 2loss/SRqqt1/assignment_loss‚1.5992557848471083 2loss/SRqqt2/assignment_loss‚2.0633042427626522 0loss/FRt4/assignment_loss‚2.0622031427919865 /loss/FBt1/detection_loss‚2.6469075456261635 epoch‚3 0loss/FBt1/assignment_loss‚0.6602505202554312 /loss/FBt2/detection_loss‚2.9328930102861843 % _timestamp‚1.7883990951734114e+09 2loss/SRqqt3/assignment_loss‚1.6917365872331205 0loss/FBt2/assignment_loss‚0.5074333615887624 .loss/FRt1/detection_loss‚2.110494031437806 _runtime‚3719.810190125 1loss/SRqqt3/detection_loss‚1.7303854888004642 /loss/FBt4/detection_loss‚0.8538149263106513 trainer/global_step‚4649 0loss/FRt3/assignment_loss‚3.2420146642847265 /loss/FBt3/assignment_loss‚0.556725399381935 .loss/FRt3/detection_loss‚2.393145941990487 1loss/SRqqt2/detection_loss‚2.5858004201542246 &loss/total_loss‚46.951785684903854 _step‚380 0loss/FRt1/assignment_loss‚3.3176131961601123 1loss/FBt4/assignment_loss‚0.11546473984591492 /loss/FRt4/detection_loss‚1.9844780859226983+<;aÙÛ 0loss/FRt1/assignment_loss‚3.3176131961601123 1loss/FBt4/assignment_loss‚0.11546473984591492 /loss/FRt4/detection_loss‚1.9844780859226983 /loss/FBt3/detection_loss‚2.4915862500667574 /loss/FRt2/detection_loss‚2.8721609357744455 0loss/FRt2/assignment_loss‚4.3684662990272045 1loss/SRqqt4/assignment_loss‚1.244445913686202 0loss/SRqqt1/detection_loss‚1.773145123251847 1loss/SRqqt4/detection_loss‚1.1480600734169666 2loss/SRqqt1/assignment_loss‚1.5992557848471083 2loss/SRqqt2/assignment_loss‚2.0633042427626522 0loss/FRt4/assignment_loss‚2.0622031427919865 /loss/FBt1/detection_loss‚2.6469075456261635 epoch‚3 0loss/FBt1/assignment_loss‚0.6602505202554312 /loss/FBt2/detection_loss‚2.9328930102861843 % _timestamp‚1.7883990951734114e+09 2loss/SRqqt3/assignment_loss‚1.6917365872331205 0loss/FBt2/assignment_loss‚0.5074333615887624 .loss/FRt1/detection_loss‚2.110494031437806 _runtime‚3719.810190125 1loss/SRqqt3/detection_loss‚1.7303854888004642 /loss/FBt4/detection_loss‚0.8538149263106513 trainer/global_step‚4649 0loss/FRt3/assignment_loss‚3.2420146642847265 /loss/FBt3/assignment_loss‚0.556725399381935 .loss/FRt3/detection_loss‚2.393145941990487 1loss/SRqqt2/detection_loss‚2.5858004201542246 &loss/total_loss‚46.951785684903854 _step‚380üõ¬¸Ú‚(:’ ÷›ãÔŒ…®® gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚127.824' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.564799999999998 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½kÿzÛ‚(:ï ÷›ãÔ©š‘± disk.drbd1716.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 7556.1328125 disk.dm-1.out‚ 7556.1328125 disk.drbd2266.in‚0 disk.sda1.out‚0 cpu‚25.317284461562835% disk./.usageGB‚226.88586807250977 disk.dm-1.in‚ 0.0546875 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 346.6796875& disk.nvme0n1.out‚23538.71923828125( proc.memory.availableMB‚ 1231.3984375 network.recv‚ 1.356461e+06 disk.drbd2080.out‚0( disk.nvme0c0n1.out‚23538.71923828125 disk.sda.in‚ 0.0546875) proc.memory.percent‚15.89508056640625 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.rbd1.in‚0$ memory_percent‚92.48414039611816 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.rbd1.out‚12.42236328125 proc.memory.rssMB‚2604.25 network.sent‚ 1.0672977e+07 disk.dm-0.in‚ 346.6796875 disk.sda3.in‚ 0.0546875 disk.sda.out‚ 7556.1328125 disk.rbd0.in‚0# disk.dm-0.out‚23538.71923828125! disk.nvme0c0n1.in‚ 346.6796875* disk./.usagePercent‚15.226500976269373„öéâ‘Ü‚(:† †œãÔ’ô® gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚111.241' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.2482 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ.4“xÝ‚(:í †œãÔì烲* disk./.usagePercent‚15.226457454988552 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda3.in‚ 0.0546875 disk.nvme0n1.in‚ 346.6796875 disk.sda.in‚ 0.0546875 disk.rbd1.out‚12.42236328125$ disk./.usageGB‚226.8852195739746 disk.sda2.in‚0 disk.dm-1.in‚ 0.0546875 disk.drbd2266.out‚0 disk.dm-0.in‚ 346.6796875 disk.sda1.out‚0 disk.sda3.out‚ 7556.72265625 network.sent‚ 1.0780999e+07 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 cpu‚25.31627218820469 disk.drbd2080.out‚0# disk.dm-0.out‚23539.21142578125 disk.sda.out‚ 7556.72265625 network.recv‚ 1.361746e+06! disk.nvme0c0n1.in‚ 346.6796875( disk.nvme0c0n1.out‚23539.21142578125$ memory_percent‚92.47856140136719% proc.memory.availableMB‚ 1232.3125 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.sda1.in‚0 proc.memory.rssMB‚2604.25) proc.memory.percent‚15.89508056640625 disk.sda2.out‚0 disk.drbd2266.in‚0& disk.nvme0n1.out‚23539.21142578125 disk.dm-1.out‚ 7556.72265625 disk.drbd2080.in‚0ÙL' Þ‚(:’ •œãÔ‘¾¢® gpu.0.gpu‚35.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚136.444' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.288799999999995 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jGTÇß‚(:ö •œãÔ߽ܰ disk.dm-1.out‚ 7557.16015625 disk.rbd1.out‚12.42236328125! disk.nvme0c0n1.in‚ 346.6796875% disk./.usageGB‚226.88525772094727& disk.nvme0n1.out‚23539.76025390625 network.recv‚ 1.37273e+06 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0 proc.cpu.threads‚17 disk.drbd2080.out‚0 network.sent‚ 1.0790295e+07* disk./.usagePercent‚15.226460015063893 disk.drbd2266.out‚0$ memory_percent‚92.48075485229492" proc.memory.rssMB‚ 2608.0859375# disk.dm-0.out‚23539.76025390625 disk.nvme0n1.in‚ 346.6796875 disk.sda2.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 1231.953125* proc.memory.percent‚15.918493270874023 cpu‚25.315232286972982 disk.rbd0.in‚0 disk.drbd2080.in‚0 disk.dm-0.in‚ 346.6796875 disk.sda1.in‚0 disk.sda.out‚ 7557.16015625 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda.in‚ 0.0546875 disk.dm-1.in‚ 0.0546875 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚ 0.0546875 disk.sda3.out‚ 7557.16015625( disk.nvme0c0n1.out‚23539.76025390625¤ïÉú2à- _wandbruntime‚3754  _runtime‚37541xaÈá trainer/global_step‚4699 % _timestamp‚1.7883991302239177e+09 _runtime‚ 3754.86077807 _step‚381 ' lr-AdamW/pg1‚0.00016487921308537797 ' lr-AdamW/pg2‚0.00016487921308537797ÜW]ÍâÇ _step‚381 ' lr-AdamW/pg1‚0.00016487921308537797 ' lr-AdamW/pg2‚0.00016487921308537797 trainer/global_step‚4699 % _timestamp‚1.7883991302239177e+09 _runtime‚ 3754.86077807ýИP°2ã- _wandbruntime‚3754  _runtime‚3754Řë Û'äÕO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 156241545 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚125326227586048 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚146336619333632 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚125326210546688 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚61530635 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚138358586789376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131744179577856 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚138358773792256 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139607119644672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚131744179577856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61530635 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6614576211456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35908606 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26531185 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6595530471570 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61530633 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚131744197580800 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94730950 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6614576211456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156261585 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62439789 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26531699 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 % _timestamp‚1.7883991302245636e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚514 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚146336847566848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚4699 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚138358773792256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚131921758057618 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚131744197580800 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6595365350166 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _step‚382 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6614407211520 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61530633 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94710912 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6614407211520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚138358586789376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94710912 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _runtime‚3754.861625809 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62440305 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204293120 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6729499688960 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚61530633 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚94710912 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6614407211520 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚138358586789376 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚94710912 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _runtime‚3754.861625809 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚62440305 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204293120 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6729499688960 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚138358586789376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚131744179577856 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚138358773792256 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚139607119644672 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚131744179577856 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_star›< V t/oversize_segments.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚61530635 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6614576211456 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚35908606 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26531185 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6595530471570 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚61530633 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚131744197580800 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚94730950 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6614576211456 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 156261585 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚62439789 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26531699 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 % _timestamp‚1.7883991302245636e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚514 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚146336847566848 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚4699 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚138358773792256 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚131921758057618 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0þœ ›}2æ- _wandbruntime‚3755  _runtime‚37554 h å&çßM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚61543294 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6596919636322 trainer/global_step‚4699 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6615969362944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚131949090404706 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚61543294 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚94750763 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚139636056532992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6730913398272 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚35915983 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚61543292 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚138387249292288 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚131771448942592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6731126066688 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _runtime‚3755.541517461 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚138387436308480 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 156294057 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156294057 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139636032592896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94730699 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6596754514710 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _step‚383 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6615800349696 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156273991 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94750763 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚138387436308480 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125352153729024 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35915981 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156273991 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26536568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62453071 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146366945991168 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚131771466945536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚131771448942592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6615969362944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125352170768384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚131948908243734 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62452549 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146367182599680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚131771466945536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6615800349696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26537088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 % _timestamp‚1.7883991309040318e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61543292 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 156294057 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚139636032592896 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚94730699 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6596754514710 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _step‚383 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6615800349696 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 156273991 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚94750763 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚138387436308480 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚125352153729024 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚35915981 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 156273991 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26536568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚62453071 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚146366945991168 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚131771466945536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚131771448942592 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6615969362944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚125352170768384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚131948908243734 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚62452549 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚146367182599680 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚131771466945536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6615800349696 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26537088 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 % _timestamp‚1.7883991309040318e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚61543292 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚131771448942592 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6731126066688 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _runtime‚3755.541517461 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944ÿy¦Æ2é- _wandbruntime‚3755  _runtime‚3755ùÖŒMWêÑ -loss/FBt1/detection_loss‚2.54538760582606 /loss/FBt4/detection_loss‚0.7069136707347898 2loss/SRqqt1/assignment_loss‚1.7939523614809982 % _timestamp‚1.7883991309058073e+09 .loss/FRt1/detection_loss‚2.203147831390489 /loss/FRt1/assignment_loss‚3.587115275409986 1loss/SRqqt2/detection_loss‚2.2997555381212478 trainer/global_step‚4699 .loss/FRt2/detection_loss‚3.296730711346581 0loss/FRt2/assignment_loss‚3.7033989372707548 2loss/SRqqt2/assignment_loss‚2.1508940144991264 2loss/SRqqt3/assignment_loss‚1.6043590691988356 /loss/FRt4/detection_loss‚2.2544599707897115 1loss/SRqqt1/detection_loss‚1.8528693792011057 1loss/SRqqt4/detection_loss‚1.0178629074132803 _step‚384 .loss/FRt3/detection_loss‚2.498759649875687 ,loss/FBt2/detection_loss‚3.3494858071208 /loss/FBt3/detection_loss‚2.6407072941462197 epoch‚3 0loss/FBt3/assignment_loss‚0.2722451970007064 0loss/FRt3/assignment_loss‚2.8444379989693807 2loss/SRqqt4/assignment_loss‚0.9777760221199556 1loss/FBt1/assignment_loss‚0.33530844861848486 0loss/FBt2/assignment_loss‚0.5049374001095828 1loss/SRqqt3/detection_loss‚1.6158431791700423 0loss/FRt4/assignment_loss‚1.8245802187338107 0loss/FBt4/assignment_loss‚0.1354909474595259 %loss/total_loss‚46.01641943600716 _runtime‚3755.542575567Œys˜\ëÖ .loss/FRt1/detection_loss‚2.203147831390489 /loss/FRt1/assignment_loss‚3.587115275409986 1loss/SRqqt2/detection_loss‚2.2997555381212478 trainer/global_step‚4699 .loss/FRt2/detection_loss‚3.296730711346581 0loss/FRt2/assignment_loss‚3.7033989372707548 2loss/SRqqt2/assignment_loss‚2.1508940144991264 2loss/SRqqt3/assignment_loss‚1.6043590691988356 /loss/FRt4/detection_loss‚2.2544599707897115 1loss/SRqqt1/detection_loss‚1.8528693792011057 1loss/SRqqt4/detection_loss‚1.0178629074132803 _step‚384 .loss/FRt3/detection_loss‚2.498759649875687 ,loss/FBt2/detection_loss‚3.3494858071208 /loss/FBt3/detection_loss‚2.6407072941462197 epoch‚3 0loss/FBt3/assignment_loss‚0.2722451970007064 0loss/FRt3/assignment_loss‚2.8444379989693807 2loss/SRqqt4/assignment_loss‚0.9777760221199556 1loss/FBt1/assignment_loss‚0.33530844861848486 0loss/FBt2/assignment_loss‚0.5049374001095828 1loss/SRqqt3/detection_loss‚1.6158431791700423 0loss/FRt4/assignment_loss‚1.8245802187338107 0loss/FBt4/assignment_loss‚0.1354909474595259 %loss/total_loss‚46.01641943600716 _runtime‚3755.542575567 -loss/FBt1/detection_loss‚2.54538760582606 /loss/FBt4/detection_loss‚0.7069136707347898 2loss/SRqqt1/assignment_loss‚1.7939523614809982 % _timestamp‚1.7883991309058073e+09€·ì‚(:’ ¤œãÔܨý­ gpu.0.gpu‚34.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚146.131' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.226200000000002 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@EéCí‚(:ô ¤œãÔý鸰% disk.nvme0n1.out‚23540.1103515625# proc.memory.rssMB‚ 2610.08203125* disk./.usagePercent‚15.226460015063893 disk.sda.in‚ 0.0546875 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 0.0546875 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0" disk.dm-0.out‚23540.1103515625 disk.sda3.in‚ 0.0546875' disk.nvme0c0n1.out‚23540.1103515625$ memory_percent‚92.47934818267822 disk.sda.out‚ 7557.7265625 disk.nvme0n1.in‚ 346.6796875 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-0.in‚ 346.6796875! disk.nvme0c0n1.in‚ 346.6796875) proc.memory.availableMB‚ 1232.18359375 network.recv‚ 1.390058e+06 disk.drbd1716.out‚0 disk.sda3.out‚ 7557.7265625% disk./.usageGB‚226.88525772094727 disk.drbd2265.in‚0 disk.rbd0.in‚0* proc.memory.percent‚15.930676460266113 cpu‚25.314120594156975 network.sent‚ 1.0802894e+07 disk.dm-1.out‚ 7557.7265625 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚12.42236328125‘ã’’î‚(:‡ ³œãÔ¾ó‡® gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚126.757' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.3514 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£ýÒ‘|ï‚(:ñ ³œãÔ‚èÿ±! disk.nvme0c0n1.in‚ 346.6796875 disk.nvme0n1.in‚ 346.6796875 disk.dm-1.in‚ 0.0546875 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚12.42236328125# proc.memory.rssMB‚ 2610.08203125 cpu‚25.313057804308766 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 disk.sda3.in‚ 0.0546875 disk.sda.in‚ 0.0546875 disk.sda.out‚ 7558.3671875 disk.dm-0.in‚ 346.6796875 disk.sda3.out‚ 7558.3671875 disk.rbd1.in‚0& disk.nvme0c0n1.out‚23541.302734375 disk.drbd1716.in‚0 disk.dm-1.out‚ 7558.3671875 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0$ memory_percent‚92.48688220977783* proc.memory.percent‚15.930676460266113! disk.dm-0.out‚23541.302734375$ disk.nvme0n1.out‚23541.302734375 network.sent‚ 1.0915186e+07 network.recv‚ 1.402872e+06* disk./.usagePercent‚15.226469231335127% disk./.usageGB‚226.88539505004883 disk.drbd2265.in‚0) proc.memory.availableMB‚ 1230.94921875X^H°2ð- _wandbruntime‚3790  _runtime‚3790¹›Ž,Éñà ' lr-AdamW/pg1‚0.00016356112464627063 ' lr-AdamW/pg2‚0.00016356112464627063 trainer/global_step‚4749 % _timestamp‚1.7883991661559012e+09 _runtime‚3790.792764549 _step‚385#äž4ÎòÈ ' lr-AdamW/pg1‚0.00016356112464627063 ' lr-AdamW/pg2‚0.00016356112464627063 trainer/global_step‚4749 % _timestamp‚1.7883991661559012e+09 _runtime‚3790.792764549 _step‚385,)Ñú2ó- _wandbruntime‚3790  _runtime‚37900ë)«/ ôÕO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62163585 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62163585 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚141052767055872 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6664837168574 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6800560420864 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚62163583 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚139791913351680 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚508 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6684247534080 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7883991661566086e+09 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚62163583 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚147853351416832 \ODeviceStatsMonitor.on_train_batch_start/inac9W랬tive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚95723652 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚510 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚147853118989312 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚133107665817600 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6665002289978 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚139791726348800 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚126623386702848 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63080407 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚133288388992826 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 _step‚386 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63079897 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26802951 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚133107647814656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36277456 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6684078534144 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26802443 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141052790995968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6800351933440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚126623369663488 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 _runtime‚3790.793657173 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95703614 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139791726348800 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6684247534080 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6665002289978 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚139791726348800 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚126623386702848 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63080407 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚133288388992826 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 _step‚386 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63079897 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚26802951 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚133107647814656 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36277456 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6684078534144 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚26802443 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚141052790995968 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6800351933440 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚126623369663488 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 _runtime‚3790.793657173 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚95703614 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚139791726348800 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6684247534080 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95743465 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚133134917179392 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 157899643 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6801764659712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62176244 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚133315539178942 $ _timestamp‚1.788399166832256e+09 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6685640685568 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157919709 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6666391454730 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 157919709 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63092545 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _step‚387 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚126649329885184 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚139820575867904 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26808220 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26807714 _runtime‚3791.469607144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚126649312845824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6685640685568 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141081680004096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62176242 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95743465 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚133315721339914 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6685471672320 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62176242 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚147883444663808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚139820575867904 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚133134935182336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚4749 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992²ŒT;øãM _step‚387 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚126649329885184 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚139820575867904 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚26808220 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚26807714 _runtime‚3791.469607144 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚126649312845824 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6685640685568 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚141081680004096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62176242 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚95743465 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚133315721339914 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6685471672320 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62176242 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚147883444663808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚139820575867904 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚133134935182336 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚4749 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6685471672320 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36284833 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚506 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚147883685466624 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚95743465 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚133134917179392 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 157899643 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ]KDeviceStatsMonitor.on_train_batchqc‡³®_end/inactive_split_bytes.small_pool.freed‚ 6801764659712 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62176244 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚133315539178942 $ _timestamp‚1.788399166832256e+09 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6685640685568 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 157919709 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6666391454730 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 157919709 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63092545 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217ƒ…—Œ2ù- _wandbruntime‚3791  _runtime‚3791r¾žUúÏ 2loss/SRqqt4/assignment_loss‚0.9924176696974497 .loss/FRt2/detection_loss‚2.750624059019862 .loss/FRt3/detection_loss‚2.343545821122825 /loss/SRqqt4/detection_loss‚1.36144905514442 _step‚388 0loss/FRt1/assignment_loss‚2.9973859308195894 /loss/FRt2/assignment_loss‚3.556482820897489 2loss/SRqqt3/assignment_loss‚1.4948681438496958 $ _timestamp‚1.788399166834006e+09 0loss/FRt4/assignment_loss‚1.5394775657092823 /loss/FRt4/detection_loss‚1.6652351210222525 /loss/FBt1/detection_loss‚3.1677874505519865 /loss/FBt3/detection_loss‚2.7933004001776376 epoch‚3 _runtime‚3791.470750923 -loss/FRt1/detection_loss‚2.14740948061474 1loss/SRqqt3/detection_loss‚1.7973082726821303 2loss/SRqqt1/assignment_loss‚1.4634456764596204 1loss/FBt4/assignment_loss‚0.15183389717603973 2loss/SRqqt2/assignment_loss‚1.8616797261767917 0loss/FBt1/assignment_loss‚0.5127436100938212 /loss/FBt3/assignment_loss‚0.587483304242293 1loss/SRqqt2/detection_loss‚2.3491101331180997 /loss/FBt4/detection_loss‚0.8959345294581338 /loss/FRt3/assignment_loss‚3.494981290400028 0loss/FBt2/assignment_loss‚0.5278122289612693 .loss/FBt2/detection_loss‚2.811689092354341 %loss/total_loss‚44.97326123039938 1loss/SRqqt1/detection_loss‚1.7092559506495795 trainer/global_step‚4749’›â+ZûÔ /loss/FBt3/assignment_loss‚0.587483304242293 1loss/SRqqt2/detection_loss‚2.3491101331180997 /loss/FBt4/detection_loss‚0.8959345294581338 /loss/FRt3/assignment_loss‚3.494981290400028 0loss/FBt2/assignment_loss‚0.5278122289612693 .loss/FBt2/detection_loss‚2.811689092354341 %loss/total_loss‚44.97326123039938 1loss/SRqqt1/detection_loss‚1.7092559506495795 trainer/global_step‚4749 2loss/SRqqt4/assignment_loss‚0.9924176696974497 .loss/FRt2/detection_loss‚2.750624059019862 .loss/FRt3/detection_loss‚2.343545821122825 /loss/SRqqt4/detection_loss‚1.36144905514442 _step‚388 0loss/FRt1/assignment_loss‚2.9973859308195894 /loss/FRt2/assignment_loss‚3.556482820897489 2loss/SRqqt3/assignment_loss‚1.4948681438496958 $ _timestamp‚1.788399166834006e+09 0loss/FRt4/assignment_loss‚1.5394775657092823 /loss/FRt4/detection_loss‚1.6652351210222525 /loss/FBt1/detection_loss‚3.1677874505519865 /loss/FBt3/detection_loss‚2.7933004001776376 epoch‚3 _runtime‚3791.470750923 -loss/FRt1/detection_loss‚2.14740948061474 1loss/SRqqt3/detection_loss‚1.7973082726821303 2loss/SRqqt1/assignment_loss‚1.4634456764596204 1loss/FBt4/assignment_loss‚0.15183389717603973 2loss/SRqqt2/assignment_loss‚1.8616797261767917 0loss/FBt1/assignment_loss‚0.5127436100938212„®¦ü‚(:’ ÂœãÔ—þ¹® gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚131.312' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.262400000000003 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬Qµ|ý‚(:ñ ÂœãԽ·± disk.nvme0n1.in‚ 346.6796875$ memory_percent‚92.48185157775879 disk.sda.out‚ 7558.91796875 disk.drbd1716.out‚0 network.sent‚ 1.0920379e+07 disk.sda.in‚ 0.0546875 disk.drbd2265.out‚0 disk.dm-0.in‚ 346.6796875 disk.sda1.out‚0 disk.rbd1.out‚12.50830078125! disk.nvme0c0n1.in‚ 346.6796875 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.sda1.in‚0* proc.memory.percent‚15.930676460266113 cpu‚25.31203249448252 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.sda3.out‚ 7558.91796875 disk.rbd1.in‚0( proc.memory.availableMB‚ 1231.7734375# proc.memory.rssMB‚ 2610.08203125% disk./.usageGB‚226.88532257080078 disk.dm-1.out‚ 7558.91796875 disk.rbd0.in‚0& disk.nvme0c0n1.out‚23541.580078125$ disk.nvme0n1.out‚23541.580078125 disk.dm-1.in‚ 0.0546875 disk.rbd0.out‚0 disk.sda3.in‚ 0.0546875 network.recv‚ 1.40577e+06 disk.sda2.in‚0! disk.dm-0.out‚23541.580078125 proc.cpu.threads‚17* disk./.usagePercent‚15.226464367191975 disk.drbd1716.in‚0 disk.drbd2080.out‚0 ®l—‘þ‚(:† ÑœãÔÚÞ’® gpu.0.gpu‚32.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚138.625' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.725 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zmvÿ‚(:ë ÑœãÔ°Çа network.sent‚ 1.10282e+07 disk.sda2.in‚0 disk.sda1.in‚0 cpu‚25.311018698991237 disk.drbd2265.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 7559.40625! disk.nvme0c0n1.in‚ 346.6796875 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda3.in‚ 0.0546875 disk.nvme0n1.in‚ 346.6796875* proc.memory.percent‚15.930676460266113* disk./.usagePercent‚15.226446190657045 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.0546875 disk.drbd2266.out‚0 disk.dm-0.in‚ 346.6796875 disk.rbd1.in‚0 disk.rbd1.out‚12.57080078125$ memory_percent‚92.48166084289551% disk./.usageGB‚226.88505172729492 disk.sda2.out‚0 disk.dm-1.out‚ 7559.40625 disk.drbd2266.in‚0" disk.dm-0.out‚23542.1474609375% disk.nvme0n1.out‚23542.1474609375( proc.memory.availableMB‚ 1231.8046875# proc.memory.rssMB‚ 2610.08203125 disk.sda.in‚ 0.0546875 disk.drbd1716.in‚0 disk.drbd2080.in‚0 proc.cpu.threads‚17 network.recv‚ 1.411839e+06 disk.sda.out‚ 7559.40625 disk.drbd2080.out‚0' disk.nvme0c0n1.out‚23542.1474609375 disk.rbd0.in‚0ÉO¥ €‚(:’ àœãÔö¦Ë® gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚136.974' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.394799999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­XÀ€‚(:õ àœãÔÙ¶Š± disk.sda2.out‚0 disk.drbd2266.in‚0$ memory_percent‚92.48378276824951 cpu‚25.309930558555585% disk./.usageGB‚226.88504791259766 disk.rbd1.in‚0 disk.nvme0n1.in‚ 346.6796875% disk.nvme0n1.out‚23542.3544921875 disk.dm-1.in‚ 0.0546875 disk.dm-1.out‚ 7560.23828125 disk.drbd2266.out‚0 disk.drbd2080.in‚0 network.sent‚ 1.1034215e+07 network.recv‚ 1.41643e+06 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0) proc.memory.availableMB‚ 1231.45703125" disk.dm-0.out‚23542.3544921875 disk.sda1.out‚0 disk.sda3.out‚ 7560.23828125 disk.rbd1.out‚12.57080078125' disk.nvme0c0n1.out‚23542.3544921875 disk.drbd2265.in‚0) disk./.usagePercent‚15.22644593464951 disk.sda3.in‚ 0.0546875! disk.nvme0c0n1.in‚ 346.6796875 disk.sda.in‚ 0.0546875 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda.out‚ 7560.23828125 disk.rbd0.in‚0 disk.dm-0.in‚ 346.6796875# proc.memory.rssMB‚ 2610.08203125* proc.memory.percent‚15.930676460266113 proc.cpu.threads‚17ÑÆ{22‚- _wandbruntime‚3826  _runtime‚3826jÞRßɃà ' lr-AdamW/pg1‚0.00016223135175796473 ' lr-AdamW/pg2‚0.00016223135175796473 trainer/global_step‚4799 % _timestamp‚1.7883992017437444e+09 _runtime‚3826.380608819 _step‚389¹4¹ˆÎ„È _step‚389 ' lr-AdamW/pg1‚0.00016223135175796473 ' lr-AdamW/pg2‚0.00016223135175796473 trainer/global_step‚4799 % _timestamp‚1.7883992017437444e+09 _runtime‚3826.380608819…*(2…- _wandbruntime‚3826  _runtime‚3826´Çu Û'†ÕO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚142498438407168 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 159512889 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚142498414467072 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚134655019928034 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚96716354 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚96716354 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚36646306 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27073023 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚141225052911104 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 trainer/global_step‚4799 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚127920528780288 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6753918856704 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6753749856768 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚141224865908224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62796535 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚127920545819648 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62796535 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚134654837767270 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚390 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63719327 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27073557 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96696316 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6871395002880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7883992017444475e+09 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63719863 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚141225052911104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6734308986982 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚134471116051456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚134471116051456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6871180224000 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚141224865908224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6753918856704 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6753749856768 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚134471134054400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚141224865908224 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚62796535 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚127920545819648 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚62796535 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚134654837767270 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _step‚390 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚63719327 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27073557 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚96696316 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6871395002880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7883992017444475e+09 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚63719863 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚141225052911104 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6734308986982 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚134471116051456 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚134471116051456 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 6871180224000 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚141224865908224 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6753918856704 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6753749856768 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚134471134054400 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159545361 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63732162 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚134498385416192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚141253528411136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96736167 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚134682170114150 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27079017 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚127946471962624 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224025600 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6872807275008 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚134682352275122 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62809192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚142527327415296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159525295 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6872607189504 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6755312008192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36653683 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚141253715427328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62809194 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36653681 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96716103 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6755142994944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 % _timestamp‚1.7883992024074552e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6755312008192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149399934604800 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142527351355392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160ýâ!Gê&ŠäM NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚62809192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚96736167 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚96716103 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚149400158630400 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27078481 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6735698151526 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚134498403419136 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 % _timestamp‚1.7883992024074552e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6755312008192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚149399934604800 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚142527351355392 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 159545361 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚63732162 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚134498385416192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚141253528411136 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚96736167 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚134682170114150 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27079017 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚127946471962624 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224025600 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6872807275008 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚134682352275122 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚62809192 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚142527327415296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 159525295 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6872607189504 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6755312008192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚36653683 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚141253715427328 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚62809194 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚36653681 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚96716103 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6755142994944 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 145311232 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 161585780 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚286 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚27513679 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2460 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚136404476650496 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2460 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚199 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚143226944359936 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 139901440 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 5041553408 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 161585780 FDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 145311232 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 161585780 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚286 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚27513679 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2460 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚136404476650496 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2460 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚199 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚143226944359936 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 139901440 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 5041553408 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 161585780 FDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚63800516 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 140964952 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 417333248 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 145311232 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 6822944153600 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2460 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚144548371829760 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 161603888 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 127308288 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚199 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚284 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 4757245257 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚143246158681600 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚136423341836288 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 5041553408 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚97819349 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚416 % _timestamp‚1.7883992771498172e+09 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚144548395769856 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚136423359839232 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 115961344 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚136578657147254 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚97803372 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 161603888 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 5041553408 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 6941378701824 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚97803372 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚136423341836288 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚982 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚27518379 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 6941262740480 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚136423341836288 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 5041553408 SFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚97819349 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚416 % _timestamp‚1.7883992771498172e+09 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚144548395769856 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocatedô¦)Ým!‚136423359839232 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 115961344 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚136578657147254 OBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚97803372 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 161603888 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 5041553408 fTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚ 6941378701824 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 SFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚97803372 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚136423341836288 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚982 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚27518379 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 6941262740480 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚63800516 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 HBDeviceStatsMonitor.on_validation_batch_end/oversize_segments.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 140964952 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 417333248 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 145311232 aODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚ 6822944153600 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2460 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚144548371829760 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944ŠJ É92ž- _wandbruntime‚3914  _runtime‚3914ñjHL^Ÿ¨¥ A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.47045302391052246 particle/accuracy_5_of_5‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21373458206653595 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5405978560447693 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3055555522441864 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9662355780601501 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.46932950615882874 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3100694715976715 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5444509387016296 7"Purity/*FRt0SRqqt3FBt/event_purity‚0.15079365670681 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5458391904830933 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9953567385673523 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_4_of_4‚0.00541973439976573 jet/accuracy_1_of_7‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5416666865348816 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5376983880996704 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.47021523118019104 jet/accuracy_4_of_6‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9822976589202881 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 +particle/sensitivity‚0.2915843427181244 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4266666769981384 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6821012496948242 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13679051399230957 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19852222502231598 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.46474969387054443 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 jet/accuracy_5_of_5‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4653683006763458 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.24132774770259857 particle/accuracy_2_of_6‚1 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/event_purity‚0.0469278059899807 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5004327297210693 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.35595235228538513 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4408300817012787 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06210761517286301 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 +particle/specificity‚0.9410244822502136 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2564815282821655 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4152505397796631 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08963683992624283 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9828906059265137 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3406068682670593 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1388888955116272 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 jet/accuracy_3_of_7‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13623134791851044 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6515151262283325 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.36837542057037354 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3219697177410126 +jet/accuracy_2_of_2‚0.10123426467180252 particle/accuracy_2_of_3‚1 _step‚395 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 particle/accuracy_1_of_6‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.699999988079071 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.05985036864876747 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3222222328186035 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09259260445833206 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9239566922187805 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.36201485991477966 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5105085372924805 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_6‚0.4615384638309479 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.47220727801322937 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5443533658981323 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9888330101966858 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26030415296554565 6 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.117325559258461 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17151419818401337 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.17746488749980927 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_4_of_4‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5692169666290283 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.38227972388267517 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5524691343307495 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.41291603446006775 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5526316165924072 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.495038241147995 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 particle/accuracy_4_of_6‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5214789509773254 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9684028029441833 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0017636684933677316 +jet/accuracy_2_of_4‚0.34354403614997864 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2579728960990906 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3832623362541199 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.39581841230392456 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4869866371154785 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13239096105098724 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13683339953422546 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.21717171370983124 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1315789520740509 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9871436953544617 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6818181872367859 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.822067141532898 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9672514200210571 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.026444384828209877 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.1875 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 particle/accuracy_3_of_5‚1 jet/accuracy_6_of_7‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.18304093182086945 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3596726059913635 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9812180399894714 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6355577707290649 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19261899590492249 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12196581810712814 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_4_of_5‚0.01036866381764412 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4159609377384186 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9815812110900879 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15025535225868225 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011688312515616417 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5292906761169434 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3654971122741699 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 'particle/f_score‚0.3618701994419098 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9753968119621277 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9811436533927917 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13797490298748016 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.565625011920929 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 epoch‚3 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18148845434188843 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4155701696872711 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.40625 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 particle/accuracy_2_of_2‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9867210984230042 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_4‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2562721073627472 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.015627361834049225 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22870387136936188 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.15000000596046448 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.33316949009895325 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0291589442640543 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006686937063932419 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4008246660232544 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12554168701171875 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5239365100860596 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29698115587234497 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_3‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.15000000596046448 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5218356847763062 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9898918867111206 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.49777597188949585 particle/accuracy_1_of_1‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06250344961881638 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1346253603696823 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4998047351837158 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4460730254650116 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2097700983285904 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2688491940498352 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.25271332263946533 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0590277798473835 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13637401163578033 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.015680043026804924 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13271592557430267 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.45708128809928894 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.48740923404693604 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20036692917346954 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5783883333206177 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.37703433632850647 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3914942741394043 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.41067543625831604 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1144995391368866 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13828644156455994 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3699180483818054 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 )jet/accuracy_1_of_3‚0.853803277015686 *jet/accuracy_1_of_6‚0.9615384340286255 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4981321692466736 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4228140711784363 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_6_of_7‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347518116235733 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3179592490196228 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13683339953422546 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15157829225063324 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5175783634185791 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.33950620889663696 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 trainer/global_step‚4835 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5188454389572144 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7337068319320679 particle/accuracy_3_of_6‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9858829379081726 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4867337644100189 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 jet/accuracy_4_of_7‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16520848870277405 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_2‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9125296473503113 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06993954628705978 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18034406006336212 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0012210012646391988 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28524860739707947 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.090409412980079Ç?b$c465 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 % _timestamp‚1.7883992897607555e+09 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9918451309204102 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5202189683914185 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4620243310928345 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.48034101724624634 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9968039393424988 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 )loss/val_total_loss‚46.59426147348893 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.4117647111415863 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.05316092073917389 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.46049901843070984 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.22635135054588318 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15542946755886078 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23198892176151276 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 particle/accuracy_4_of_7‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11401897668838501 *jet/accuracy_1_of_4‚0.8909760117530823 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9882231950759888 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4279070794582367 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.45194289088249207 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.16896551847457886 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.45372357964515686 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.35522139072418213 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_7‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5416666865348816 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_7‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4721880257129669 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16524063050746918 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9697389006614685 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21045920252799988 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9929183721542358 jet/accuracy_6_of_6‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03162623941898346 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 4 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1350087672472 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17978718876838684 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.27897897362709045 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2799662947654724 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.47602933645248413 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.023218009620904922 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4597851634025574 *"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.369148313999176 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9710884690284729 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.48961231112480164 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08138731122016907 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13649627566337585 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3645833134651184 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.18939393758773804 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.016830546781420708 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.36918604373931885 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05550595372915268 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3576388955116272 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5133333802223206 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19717735052108765 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.021661655977368355 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9773584604263306 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1388888955116272 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964687824249268 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9864603877067566 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.0244904775172472 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1149425357580185 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3916666805744171 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 (particle/accuracy‚0.8396164178848267 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1763659566640854 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1788446605205536 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5718390941619873 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_7‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.10000000149011612 particle/accuracy_5_of_6‚1 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4681974947452545 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5369064211845398 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 * Purity/3FRt1SRqqt1FBt/FRt_purity‚0.375 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.48961231112480164 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9967366456985474 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 +jet/accuracy_3_of_6‚0.19230769574642181 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5099294781684875 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 particle/accuracy_2_of_4‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2384933978319168 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.31604939699172974 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21235373616218567 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5130046606063843 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 _runtime‚3914.399624417 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 +jet/accuracy_3_of_5‚0.10046722739934921 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5205357074737549 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3055555522441864 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4865279495716095 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9904718399047852 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735631346702576ëlúK ­¥ particle/accuracy_6_of_6‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1388888955116272 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 jet/accuracy_3_of_7‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13623134791851044 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6515151262283325 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.36837542057037354 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3219697177410126 +jet/accuracy_2_of_2‚0.10123426467180252 particle/accuracy_2_of_3‚1 _step‚395 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 particle/accuracy_1_of_6‚1 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.699999988079071 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.05985036864876747 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3222222328186035 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ("Purity/2FRt*SRqqt2FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09259260445833206 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9239566922187805 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.36201485991477966 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5105085372924805 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_6‚0.4615384638309479 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9285715222358704 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.47220727801322937 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5443533658981323 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9888330101966858 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26030415296554565 6 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.117325559258461 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.28125 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.17151419818401337 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.17746488749980927 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 particle/accuracy_4_of_4‚1 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5692169666290283 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.38227972388267517 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5524691343307495 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.41291603446006775 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5526316165924072 8"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.495038241147995 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 particle/accuracy_4_of_6‚1 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5214789509773254 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9684028029441833 <"Purity/2FRt0SRqqt0FBt/event_purity‚0.0017636684933677316 +jet/accuracy_2_of_4‚0.34354403614997864 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.2579728960990906 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.3832623362541199 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.39581841230392456 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.4869866371154785 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13239096105098724 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.13683339953422546 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.21717171370983124 9"Purity/0FRt*SRqqt3FBt/event_purity‚0.1315789520740509 & Purity/1FRt2SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9871436953544617 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 ("Purity/2FRt0SRqqt2FBt/event_purity‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.6818181872367859 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.2777777910232544 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.822067141532898 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9672514200210571 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.5714285969734192 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.026444384828209877 -"Purity/0FRt0SRqqt3FBt/event_purity‚0.1875 ( Purity/1FRt1SRqqt3FBt/FRt_purity‚0.5 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 particle/accuracy_3_of_5‚1 jet/accuracy_6_of_7‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.18304093182086945 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3596726059913635 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9812180399894714 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833134651184 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6355577707290649 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19261899590492249 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12196581810712814 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_4_of_5‚0.01036866381764412 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4159609377384186 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9815812110900879 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15025535225868225 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9929742813110352 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011688312515616417 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5292906761169434 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3654971122741699 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 'particle/f_score‚0.3618701994419098 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9753968119621277 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9811436533927917 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13797490298748016 8"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.565625011920929 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 epoch‚3 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18148845434188843 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4155701696872711 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.40625 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 particle/accuracy_2_of_2‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9867210984230042 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.0555555559694767 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 particle/accuracy_3_of_4‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2562721073627472 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.015627361834049225 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.22870387136936188 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.15000000596046448 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.33316949009895325 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0291589442640543 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.006686937063932419 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4008246660232544 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12554168701171875 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9833333492279053 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5239365100860596 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.29698115587234497 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_3‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.15000000596046448 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5218356847763062 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9898918867111206 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.49777597188949585 particle/accuracy_1_of_1‚1 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.06250344961881638 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1346253603696823 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.4998047351837158 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4460730254650116 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2097700983285904 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2688491940498352 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.25271332263946533 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/1FRt1SRqqt2FBt/event_purity‚0.0590277798473835 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13637401163578033 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.015680043026804924 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13271592557430267 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 :"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.45708128809928894 :"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.48740923404693604 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20036692917346954 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5783883333206177 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.37703433632850647 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3914942741394043 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.41067543625831604 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ,"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.375 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1144995391368866 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.13828644156455994 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.3699180483818054 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 )jet/accuracy_1_of_3‚0.853803277015686 *jet/accuracy_1_of_6‚0.9615384340286255 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4981321692466736 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 6validation_average_jet_accuracy‚0.4228140711784363 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_6_of_7‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.11347518116235733 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3179592490196228 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.13683339953422546 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15157829225063324 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5175783634185791 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.33950620889663696 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 particle/accuracy_2_of_7‚1 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 trainer/global_step‚4835 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5188454389572144 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_2‚0.7337068319320679 particle/accuracy_3_of_6‚1 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9858829379081726 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.4867337644100189 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 jet/accuracy_4_of_7‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.16520848870277405 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_2‚1 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9125296473503113 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.06993954628705978 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18034406006336212 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0012210012646391988 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.28524860739707947 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.09040941298007965 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 % _timestamp‚1.7883992897607555e+09 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9918451309204102 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5202189683914185 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4620243310928345 ?&Purity/3FRt*SRqqt0FBt/event_propÛÈé“%Gortion‚0.007254464086145163 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 :"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.48034101724624634 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9968039393424988 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 )loss/val_total_loss‚46.59426147348893 7 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.4117647111415863 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.05316092073917389 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.46049901843070984 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.22635135054588318 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15542946755886078 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23198892176151276 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 particle/accuracy_4_of_7‚1 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11401897668838501 *jet/accuracy_1_of_4‚0.8909760117530823 ( Purity/*FRt1SRqqt3FBt/FRt_purity‚0.5 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9882231950759888 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4279070794582367 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.45194289088249207 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.16896551847457886 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.45372357964515686 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.35522139072418213 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_7‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5416666865348816 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_7‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.4721880257129669 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.16524063050746918 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9697389006614685 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.7222222089767456 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21045920252799988 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9929183721542358 jet/accuracy_6_of_6‚0 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.03162623941898346 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 4 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1350087672472 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17978718876838684 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.27897897362709045 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.2799662947654724 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.47602933645248413 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.023218009620904922 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.4597851634025574 *"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.369148313999176 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9710884690284729 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.48961231112480164 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08138731122016907 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_7_of_7‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13649627566337585 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3645833134651184 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.18939393758773804 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 -"Purity/1FRt0SRqqt3FBt/event_purity‚0.0625 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.016830546781420708 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.36918604373931885 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05550595372915268 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.3576388955116272 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5133333802223206 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.19717735052108765 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.021661655977368355 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9773584604263306 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1388888955116272 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964687824249268 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9864603877067566 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9351851940155029 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 9"Purity/*FRt2SRqqt*FBt/event_purity‚0.0244904775172472 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 7 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.1149425357580185 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3916666805744171 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 (particle/accuracy‚0.8396164178848267 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1763659566640854 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1788446605205536 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5718390941619873 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_7‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.10000000149011612 particle/accuracy_5_of_6‚1 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4681974947452545 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5369064211845398 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 * Purity/3FRt1SRqqt1FBt/FRt_purity‚0.375 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.48961231112480164 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9967366456985474 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 +jet/accuracy_3_of_6‚0.19230769574642181 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5099294781684875 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.5625 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 particle/accuracy_2_of_4‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2384933978319168 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.31604939699172974 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.21235373616218567 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5130046606063843 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 _runtime‚3914.399624417 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9967948794364929 +jet/accuracy_3_of_5‚0.10046722739934921 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 + Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9375 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5205357074737549 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3055555522441864 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4865279495716095 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9904718399047852 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735631346702576 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.2666666805744171 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.47045302391052246 particle/accuracy_5_of_5‚1 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.21373458206653595 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5405978560447693 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3055555522441864 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9662355780601501 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.46932950615882874 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3100694715976715 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.375 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5444509387016296 7"Purity/*FRt0SRqqt3FBt/event_purity‚0.15079365670681 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5458391904830933 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9953567385673523 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 +jet/accuracy_4_of_4‚0.00541973439976573 jet/accuracy_1_of_7‚1 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5416666865348816 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5376983880996704 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.47021523118019104 jet/accuracy_4_of_6‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9822976589202881 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 +particle/sensitivity‚0.2915843427181244 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4266666769981384 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6821012496948242 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.13679051399230957 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.19852222502231598 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.46474969387054443 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 jet/accuracy_5_of_5‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4653683006763458 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 +jet/accuracy_2_of_3‚0.24132774770259857 particle/accuracy_2_of_6‚1 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/event_purity‚0.0469278059899807 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_7‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5004327297210693 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.35595235228538513 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4408300817012787 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06210761517286301 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 +particle/specificity‚0.9410244822502136 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2564815282821655 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4152505397796631 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.08963683992624283 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9828906059265137 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3406068682670593 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0‹îJø‘¡‚(:† ºãÔ²†® gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚122.296' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4592 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0´}¢‚(:ò ºãÔ‰±–² disk.sda2.in‚0 cpu‚25.38323061829233 network.recv‚ 1.443087e+06 disk.sda2.out‚0 disk.dm-0.in‚ 346.6796875 disk.sda1.in‚0! proc.memory.rssMB‚ 2694.515625 disk.sda3.out‚ 7564.2734375) disk./.usagePercent‚15.13738014546471 disk.sda.out‚ 7564.2734375 disk.dm-1.in‚ 0.0546875 disk.dm-1.out‚ 7564.2734375 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0% disk./.usageGB‚225.55790328979492& disk.nvme0n1.out‚23548.52001953125 disk.drbd2266.out‚0 disk.rbd0.in‚0 network.sent‚ 1.1172018e+07 disk.nvme0n1.in‚ 346.6796875 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚23548.52001953125 disk.sda3.in‚ 0.0546875 disk.sda.in‚ 0.0546875 disk.drbd1716.out‚0 disk.rbd0.out‚0# disk.dm-0.out‚23548.52001953125$ memory_percent‚92.98584461212158) proc.memory.availableMB‚ 1149.19921875) proc.memory.percent‚16.44601821899414 disk.sda1.out‚0 disk.drbd2080.in‚0! disk.nvme0c0n1.in‚ 346.6796875 proc.cpu.threads‚17 disk.rbd1.out‚12.72900390625ŽYÇA2£- _wandbruntime‚3926  _runtime‚3926ÍŽǤÁ ' lr-AdamW/pg1‚0.00016089017154313713 ' lr-AdamW/pg2‚0.00016089017154313713 trainer/global_step‚4849 # _timestamp‚1.78839930174817e+09 _runtime‚3926.384994628 _step‚396«®]?Ì¥Æ ' lr-AdamW/pg1‚0.00016089017154313713 ' lr-AdamW/pg2‚0.00016089017154313713 trainer/global_step‚4849 # _timestamp‚1.78839930174817e+09 _runtime‚3926.384994628 _step‚396ŒÀlž2¦- _wandbruntime‚3926  _runtime‚3926AuÞÛ'§ÕO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 % _timestamp‚1.7883993017488763e+09 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65075363 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚152143312882176 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162261242 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _step‚397 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64104339 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137023108996096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98156903 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98176941 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚143868711726080 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152143553698304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6964443249152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚137171365050620 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚130345485777920 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6825862233340 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64104339 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37424654 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚507 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6845602729984 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6845433730048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64104341 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98156903 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚143868524723200 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162281282 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚130345502817280 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6845433730048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚143868711726080 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚143868524723200 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 trainer/global_step‚4849 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚507 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6845602729984 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6845433730048 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚64104341 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 PCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚98156903 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚143868524723200 N@×¾<$DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 162281282 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚130345502817280 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6845433730048 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚143868711726080 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚143868524723200 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 trainer/global_step‚4849 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 162261242 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _step‚397 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64104339 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚137023108996096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚98156903 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚98176941 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚143868711726080 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚152143553698304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 6964443249152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚137171365050620 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚130345485777920 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6825862233340 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64104339 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37424654 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152173878206464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6965854809088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚137198697397708 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37432031 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64117000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65087404 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚137050378360832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64116998 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98196754 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚130371428960256 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚137050360357888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚137050360357888 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6965640043520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _step‚398 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98196754 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162293688 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27655373 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37432033 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4849 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162313754 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚491 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6846826868224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98176690 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚143897187226112 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137050378360832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6846995881472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27655862 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416Ç(šê&«äM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6846826868224 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚98176690 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚143897187226112 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚137050378360832 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6846995881472 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27655862 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚64117000 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚145207999457280 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚143897374242304 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚152173878206464 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 6965854809088 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚137198697397708 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚37432031 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64117000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65087404 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚137050378360832 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64116998 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚98196754 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚130371428960256 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚137050360357888 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚137050360357888 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 6965640043520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _step‚398 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚98196754 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 162293688 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚27655373 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚37432033 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚4849 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 162313754 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚491 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408Ž»þ*è2¬- _wandbruntime‚3927  _runtime‚3927?a0PU­Ï 1loss/SRqqt2/detection_loss‚2.6055312022086112 /loss/FBt2/assignment_loss‚1.101246608154968 .loss/FRt2/detection_loss‚3.196409544119468 trainer/global_step‚4849 _runtime‚3927.062000538 /loss/FRt3/assignment_loss‚3.161773448898679 1loss/FBt4/assignment_loss‚0.29773698836792845 /loss/FRt4/detection_loss‚1.7480997397911315 0loss/SRqqt3/detection_loss‚2.264672302827239 $ _timestamp‚1.788399302425268e+09 _step‚399 2loss/SRqqt1/assignment_loss‚1.7529255214723802 .loss/FBt4/detection_loss‚0.742018171657454 /loss/FRt1/assignment_loss‚3.910842693514294 1loss/SRqqt4/detection_loss‚1.0031096343199413 /loss/FBt3/detection_loss‚3.0238038642065868 %loss/total_loss‚48.53422488822188 1loss/FBt3/assignment_loss‚0.36840761871202893 /loss/FBt1/detection_loss‚1.9502198100090027 .loss/FBt2/detection_loss‚3.620226093700954 0loss/FRt4/assignment_loss‚1.9640976365874796 .loss/FRt3/detection_loss‚2.339991106163888 epoch‚4 /loss/FRt2/assignment_loss‚4.492503588016216 2loss/SRqqt2/assignment_loss‚1.9560512015896458 /loss/FRt1/detection_loss‚2.2073498837373875 2loss/SRqqt3/assignment_loss‚1.5761514407582582 2loss/SRqqt4/assignment_loss‚1.0124377493808667 .loss/FBt1/assignment_loss‚0.31323777551118 1loss/SRqqt1/detection_loss‚1.9253812645162856;ôÙÆZ®Ô /loss/FRt1/assignment_loss‚3.910842693514294 1loss/SRqqt4/detection_loss‚1.0031096343199413 /loss/FBt3/detection_loss‚3.0238038642065868 %loss/total_loss‚48.53422488822188 1loss/FBt3/assignment_loss‚0.36840761871202893 /loss/FBt1/detection_loss‚1.9502198100090027 .loss/FBt2/detection_loss‚3.620226093700954 0loss/FRt4/assignment_loss‚1.9640976365874796 .loss/FRt3/detection_loss‚2.339991106163888 epoch‚4 /loss/FRt2/assignment_loss‚4.492503588016216 2loss/SRqqt2/assignment_loss‚1.9560512015896458 /loss/FRt1/detection_loss‚2.2073498837373875 2loss/SRqqt3/assignment_loss‚1.5761514407582582 2loss/SRqqt4/assignment_loss‚1.0124377493808667 .loss/FBt1/assignment_loss‚0.31323777551118 1loss/SRqqt1/detection_loss‚1.9253812645162856 1loss/SRqqt2/detection_loss‚2.6055312022086112 /loss/FBt2/assignment_loss‚1.101246608154968 .loss/FRt2/detection_loss‚3.196409544119468 trainer/global_step‚4849 _runtime‚3927.062000538 /loss/FRt3/assignment_loss‚3.161773448898679 1loss/FBt4/assignment_loss‚0.29773698836792845 /loss/FRt4/detection_loss‚1.7480997397911315 0loss/SRqqt3/detection_loss‚2.264672302827239 $ _timestamp‚1.788399302425268e+09 _step‚399 2loss/SRqqt1/assignment_loss‚1.7529255214723802 .loss/FBt4/detection_loss‚0.742018171657454kRI?¯‚(:’ ÉãÔå°® gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚135.089' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.017799999999998 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0W€èz°‚(:ï ÉãԬΉ± network.sent‚ 1.1317463e+07 disk.sda1.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚23683.98095703125 disk.dm-1.out‚ 7564.7421875 proc.memory.rssMB‚ 2693.53125) proc.memory.percent‚16.44001007080078 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.sda3.in‚ 0.0546875 disk.sda3.out‚ 7564.7421875 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda.in‚ 0.0546875 disk.sda2.out‚0& proc.memory.availableMB‚ 1150.71875 disk.drbd2080.in‚0 disk.sda.out‚ 7564.7421875 disk.sda2.in‚0) disk./.usagePercent‚15.13738065747978 disk.dm-0.in‚ 346.6796875 disk.rbd1.out‚12.87744140625 disk.nvme0n1.in‚ 346.6796875 disk.drbd1716.in‚0 cpu‚25.381854977213795 network.recv‚ 1.449735e+06& disk.nvme0n1.out‚23683.98095703125 disk.dm-1.in‚ 0.0546875 disk.drbd2265.out‚0% disk./.usageGB‚225.55791091918945# disk.dm-0.out‚23683.98095703125 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 346.6796875 disk.drbd1716.out‚0$ memory_percent‚92.97657012939453 ±‚(:’ ØãÔÚ‚«® gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚131.064' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.212799999999998 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘Ó–k€²‚(:õ ØãÔÁü±* disk./.usagePercent‚15.137383473562654 disk.sda3.in‚ 0.0546875 disk.rbd1.out‚12.87744140625 disk.nvme0n1.in‚ 346.6796875 disk.drbd2266.out‚0 network.sent‚ 1.1425499e+07 network.recv‚ 1.455658e+06 disk.rbd1.in‚0 disk.drbd2265.out‚0# disk.dm-0.out‚23795.12158203125( proc.memory.availableMB‚ 1233.8359375 cpu‚25.380589266797067( disk.nvme0c0n1.out‚23795.12158203125 disk.dm-1.in‚ 0.0546875 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.dm-0.in‚ 346.6796875 disk.sda.in‚ 0.0546875 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.sda1.out‚0) proc.memory.percent‚16.44001007080078 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0% disk./.usageGB‚225.55795288085938! disk.nvme0c0n1.in‚ 346.6796875& disk.nvme0n1.out‚23795.12158203125 disk.dm-1.out‚ 7565.76953125$ memory_percent‚92.46926307678223 disk.sda3.out‚ 7565.76953125 disk.sda.out‚ 7565.76953125 disk.sda1.in‚0 proc.memory.rssMB‚ 2693.53125ôu/X³‚(:’ çãÔ¬ÆÀ® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚117.311' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.462200000000003 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™J’h´‚(:Ý çãÔ銱 disk.rbd0.out‚0 disk.dm-0.out‚ 23796.2421875 disk.sda3.in‚ 0.0546875" disk.nvme0n1.out‚ 23796.2421875 disk.drbd1716.out‚0 network.sent‚ 1.1431787e+07$ disk.nvme0c0n1.out‚ 23796.2421875 disk.sda.in‚ 0.0546875 disk.drbd2266.in‚0 disk.drbd2265.out‚0! disk.nvme0c0n1.in‚ 346.6796875 disk.drbd2266.out‚0 proc.memory.rssMB‚ 2693.53125* disk./.usagePercent‚15.137387313675669 disk.dm-0.in‚ 346.6796875 disk.sda1.out‚0 disk.rbd1.out‚12.90966796875 disk.sda1.in‚0 disk.dm-1.in‚ 0.0546875 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.sda3.out‚ 7566.3125 disk.rbd1.in‚0 disk.sda.out‚ 7566.3125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 7566.3125% disk./.usageGB‚225.55801010131836 disk.rbd0.in‚0 disk.drbd1716.in‚0$ memory_percent‚93.02678108215332( proc.memory.availableMB‚ 1142.4921875 cpu‚25.379293864914153 network.recv‚ 1.460471e+06 disk.nvme0n1.in‚ 346.6796875) proc.memory.percent‚16.44001007080078 disk.drbd2080.in‚0 disk.drbd2080.out‚0ä…_s2µ- _wandbruntime‚3962  _runtime‚3962ÑÀÞäȶ % _timestamp‚1.7883993377728024e+09 _runtime‚ 3962.40964582 _step‚400 ' lr-AdamW/pg1‚0.00015953786350173435 ' lr-AdamW/pg2‚0.00015953786350173435 trainer/global_step‚4899»±Í·Ç _step‚400 ' lr-AdamW/pg1‚0.00015953786350173435 ' lr-AdamW/pg2‚0.00015953786350173435 trainer/global_step‚4899 % _timestamp‚1.7883993377728024e+09 _runtime‚ 3962.40964582fq 2¸- _wandbruntime‚3962  _runtime‚3962pé½ÿÚ'¹ÔO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7035301633536 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6895168930344 PCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚99169643 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚145301851285504 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚146624733920256 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚138537813825064 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚131642661934080 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚64737289 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6915274052608 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚138537995985828 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6915105052672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65714408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27920369 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚153659831260672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 163886894 _runtime‚3962.410557781 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 $ _timestamp‚1.788399337773493e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99149605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7035097340416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚145301664282624 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚64737289 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚65713873 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚146624757860352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27920902 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 163906934 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99169643 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚145301664282624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚138386559229952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37793504 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6895334051748 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚27920902 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 163906934 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚99169643 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚145301664282624 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚138386559229952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚37793504 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6895334051748 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6915105052672 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚65714408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚27920369 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚153659831260672 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 163886894 _runtime‚3962.410557781 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 $ _timestamp‚1.788399337773493e+09 L?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚99149605 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7035097340416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚401‘ êU2»- _wandbruntime‚3963  _runtime‚3963pOÇJ#¼ÞM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚145330513801728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 6896723216500 $ _timestamp‚1.788399338452298e+09 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163919340 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64749948 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 trainer/global_step‚4899 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚145330513801728 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _runtime‚3963.089588926 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 163919340 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7036515305984 _step‚402 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚153690396685824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚138565328332916 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚138413846597632 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚145330326785536 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99189456 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65727179 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65726655 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64749948 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚153690162174464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99169392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6916498190848 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚131668605116416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64749950 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163939406 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99189456 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27926296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163939406 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 163919340 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7036515305984 _step‚402 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚153690396685824 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚138565328332916 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚138413846597632 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚145330326785536 REDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚99189456 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚65727179 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚65726655 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚64749948 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚153690162174464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚99169392 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6916498190848 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚131668605116416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚64749950 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 163939406 NADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚99189456 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚27926296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 163939406 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 163919340 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚64749948 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 trainer/global_step‚4899 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2’^”AŠ2¾- _wandbruntime‚3963  _runtime‚3963”Y£Y¿Ó 0loss/FRt2/assignment_loss‚3.7963887538228716 0loss/FBt3/assignment_loss‚0.5296856339521128 /loss/FRt1/detection_loss‚1.9483385185406215 /loss/FRt3/detection_loss‚2.8990009715159735 0loss/SRqqt4/detection_loss‚1.079716862154168 /loss/FBt2/detection_loss‚2.7705792912415097 0loss/SRqqt3/detection_loss‚1.863114990790685 _step‚403 /loss/FRt2/detection_loss‚2.9568989042724882 .loss/FRt4/detection_loss‚1.858354565655363 1loss/SRqqt1/detection_loss‚1.9047747275163962 /loss/FBt4/detection_loss‚0.7259384942679683 &loss/total_loss‚45.769520848838496 0loss/FRt3/assignment_loss‚2.2644650955994923 1loss/FBt4/assignment_loss‚0.29072371150112025 0loss/SRqqt2/detection_loss‚2.325738852674311 /loss/FRt4/assignment_loss‚2.011937198967769 2loss/SRqqt3/assignment_loss‚1.9731717566235198 /loss/FBt3/detection_loss‚2.6638349443674088 _runtime‚3963.090656049 trainer/global_step‚4899 1loss/SRqqt4/assignment_loss‚1.100455678344981 0loss/FBt1/assignment_loss‚0.5301688702557436 0loss/FBt2/assignment_loss‚0.3162306256681729 /loss/FBt1/detection_loss‚2.3734340535269842 epoch‚4 $ _timestamp‚1.788399338453912e+09 /loss/FRt1/assignment_loss‚3.950081452175423 2loss/SRqqt1/assignment_loss‚1.5347210153244262 2loss/SRqqt2/assignment_loss‚2.1017658800789802'ÆTF^ÀØ /loss/FRt2/detection_loss‚2.9568989042724882 .loss/FRt4/detection_loss‚1.858354565655363 1loss/SRqqt1/detection_loss‚1.9047747275163962 /loss/FBt4/detection_loss‚0.7259384942679683 &loss/total_loss‚45.769520848838496 0loss/FRt3/assignment_loss‚2.2644650955994923 1loss/FBt4/assignment_loss‚0.29072371150112025 0loss/SRqqt2/detection_loss‚2.325738852674311 /loss/FRt4/assignment_loss‚2.011937198967769 2loss/SRqqt3/assignment_loss‚1.9731717566235198 /loss/FBt3/detection_loss‚2.6638349443674088 _runtime‚3963.090656049 trainer/global_step‚4899 1loss/SRqqt4/assignment_loss‚1.100455678344981 0loss/FBt1/assignment_loss‚0.5301688702557436 0loss/FBt2/assignment_loss‚0.3162306256681729 /loss/FBt1/detection_loss‚2.3734340535269842 epoch‚4 $ _timestamp‚1.788399338453912e+09 /loss/FRt1/assignment_loss‚3.950081452175423 2loss/SRqqt1/assignment_loss‚1.5347210153244262 2loss/SRqqt2/assignment_loss‚2.1017658800789802 0loss/FRt2/assignment_loss‚3.7963887538228716 0loss/FBt3/assignment_loss‚0.5296856339521128 /loss/FRt1/detection_loss‚1.9483385185406215 /loss/FRt3/detection_loss‚2.8990009715159735 0loss/SRqqt4/detection_loss‚1.079716862154168 /loss/FBt2/detection_loss‚2.7705792912415097 0loss/SRqqt3/detection_loss‚1.863114990790685 _step‚403“xä‘Á‚(:† öãÔê’–® gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚139.685' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.937 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒÞŽd‚(:Ù öãÔ­÷í° disk.drbd2266.in‚0$ memory_percent‚93.61610412597656 disk.dm-0.in‚ 346.6796875 disk.sda1.out‚0 network.sent‚ 1.1437255e+07 disk.drbd2266.out‚0 disk.nvme0n1.in‚ 346.6796875 disk.sda.in‚ 0.0546875 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚7566.875 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda3.out‚7566.875! disk.nvme0c0n1.in‚ 346.6796875% disk.nvme0c0n1.out‚23796.53515625# disk.nvme0n1.out‚23796.53515625% disk./.usageGB‚225.55797576904297 disk.dm-1.in‚ 0.0546875) disk./.usagePercent‚15.13738500960786 disk.sda1.in‚0 disk.rbd1.out‚12.90966796875 proc.memory.rssMB‚ 2693.53125 cpu‚25.378061962472337 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda.out‚7566.875 disk.drbd1716.in‚0% proc.memory.availableMB‚ 1045.9375) proc.memory.percent‚16.44001007080078 proc.cpu.threads‚17 network.recv‚ 1.463987e+06 disk.drbd2265.out‚0 disk.dm-0.out‚23796.53515625 disk.sda3.in‚ 0.0546875 disk.rbd1.in‚0 disk.drbd1716.out‚0iž2Â(:’ …žãÔÓ飮 gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚117.841' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.568199999999997 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0è+.tvÄ‚(:ë …žãÔÄÔê°) proc.memory.availableMB‚ 1026.50390625) proc.memory.percent‚16.44001007080078 cpu‚25.376836672182055 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.sda3.out‚ 7567.26953125 proc.memory.rssMB‚ 2693.53125$ disk./.usageGB‚225.5580177307129 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0$ disk.nvme0n1.out‚23797.119140625* disk./.usagePercent‚15.137387825690737 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.sda.in‚ 0.0546875 proc.cpu.threads‚17 disk.rbd0.in‚0$ memory_percent‚93.73471736907959 network.sent‚ 1.154556e+07 disk.sda2.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 network.recv‚ 1.470044e+06 disk.dm-1.out‚ 7567.26953125 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.dm-0.in‚ 346.6796875! disk.dm-0.out‚23797.119140625! disk.nvme0c0n1.in‚ 346.6796875& disk.nvme0c0n1.out‚23797.119140625 disk.dm-1.in‚ 0.0546875 disk.sda3.in‚ 0.0546875 disk.rbd1.out‚ 12.97265625 disk.nvme0n1.in‚ 346.6796875 disk.sda.out‚ 7567.26953125ÔFÏr2Å- _wandbruntime‚3998  _runtime‚3998ÁšÉÆÆÀ trainer/global_step‚4949 % _timestamp‚1.7883993739918306e+09 _runtime‚ 3998.62867558 _step‚404 & lr-AdamW/pg1‚0.0001581747094527253 & lr-AdamW/pg2‚0.0001581747094527253<Ê­DËÇÅ & lr-AdamW/pg1‚0.0001581747094527253 & lr-AdamW/pg2‚0.0001581747094527253 trainer/global_step‚4949 % _timestamp‚1.7883993739918306e+09 _runtime‚ 3998.62867558 _step‚404”V²ƒ 2È- _wandbruntime‚3998  _runtime‚3998}€æß'ÉÙO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚146734803842048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚155176277424640 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚132939821050880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6984945375232 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28194553 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6964640748752 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6984776375296 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100162345 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 100142307 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165512546 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6984776375296 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚139750027466752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38162356 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚139904444760272 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚139750045469696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚132939804011520 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100162345 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100142307 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38162354 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65370241 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚559 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139904626921036 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚146734990844928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66356909 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6964805870156 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚139750027466752 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚146734990844928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28193994 % _timestamp‚1.7883993739925504e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6984945375232 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚561 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7106108774912 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚155176514046464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚139750045469696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚148070405271552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7105896093184 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165532586 _runtime‚3998.629655459 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚405 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚146734803842048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚653702390ªXôOÊÞO YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚139904626921036 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚146734990844928 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚66356909 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 6964805870156 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚139750027466752 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚146734990844928 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28193994 % _timestamp‚1.7883993739925504e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 6984945375232 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚561 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7106108774912 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚155176514046464 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚139750045469696 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚148070405271552 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7105896093184 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 165532586 _runtime‚3998.629655459 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚405 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚146734803842048 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚65370239 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚146734803842048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚155176277424640 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚132939821050880 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 6984945375232 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28194553 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 6964640748752 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 6984776375296 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 100162345 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 10014zù!•2307 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 165512546 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 6984776375296 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚139750027466752 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38162356 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚139904444760272 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚139750045469696 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚132939804011520 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 100162345 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 100142307 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38162354 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚65370241 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚559 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128•òÎzT2Ë- _wandbruntime‚3999  _runtime‚3999Úìdé&ÌãM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165565058 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7883993746708038e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165565058 _step‚406 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚132965747193856 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165544992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚146763653361152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65382898 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6986169513472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚146763653361152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6986169513472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6986338526720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65382900 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66369620 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100162094 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚533 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7107527076864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100182158 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚146763466344960 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚155206845296640 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚132965764233216 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚155206612882432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139931777107152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _runtime‚3999.308524593 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6966029913296 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38169731 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚139777314834432 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚139777314834432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66369085 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚139777296831488 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚148099294279680 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6986338526720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38169733 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7107318602752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165544992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28199887 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0~Õ¶*î&ÍèM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚65382898 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 100162094 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚65382900 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚66369085 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚139777296831488 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚148099294279680 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 6986338526720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38169733 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7107318602752 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 165544992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28199887 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 165565058 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 % _timestamp‚1.7883993746708038e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 165565058 _step‚406 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚132965747193856 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 165544992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚146763653361152 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚65382898 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 6986169513472 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚146763653361152 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 6986169513472 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 6986338526720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚65382900 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚66369620 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 100162094 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚533 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7107527076864 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 100182158 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚146763466344960 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚155206845296640 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚132965764233216 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚155206612882432 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚139931777107152 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _runtime‚3999.308524593 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 6966029913296 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38169731 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚139777314834432 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚139777314834432 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148167943401472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38531206 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28468216 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚511 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚149516028742656 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚148168130404352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101155047 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚156693067023872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66003191 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101135009 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚141271075695480 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66998909 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚148168130404352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7054616697856 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚141113495703552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚134236963128320 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _runtime‚4034.434049505 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 $ _timestamp‚1.788399409797036e+09 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66003191 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚141113513706496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7054616697856 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7034112567160 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚409 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚141113513706496 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66003189 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202195968 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 167138198 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28467705 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚148167943401472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7034277688564 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚141113495703552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101135009 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7177014341120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38531204 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66003189 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7054447697920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28467705 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚148167943401472 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7034277688564 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚141113495703552 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 101135009 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7177014341120 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚38531204 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚66003189 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7054447697920 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚148167943401472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38531206 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚28468216 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚511 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚149516028742656 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚148168130404352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 101155047 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚156693067023872 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚66003191 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 101135009 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚141271075695480 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚66998909 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚148168130404352 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7054616697856 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚141113495703552 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416™XŒCÛ‚(:’ ²žãÔ¿³‹° gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚131.312' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.262400000000003 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0×3Հ܂(:õ ²žãÔø‘¶± network.sent‚ 1.1665599e+07 network.recv‚ 1.484648e+06 disk.rbd0.in‚0 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 7569.02734375& disk.nvme0n1.out‚23798.39306640625) proc.memory.availableMB‚ 1023.85546875* disk./.usagePercent‚15.137452595596901# disk.dm-0.out‚23798.39306640625 disk.rbd1.out‚13.03564453125) proc.memory.percent‚16.44001007080078 disk.drbd2080.in‚0 disk.sda3.in‚ 0.0546875! disk.nvme0c0n1.in‚ 346.6796875( disk.nvme0c0n1.out‚23798.39306640625$ memory_percent‚93.75088214874268 proc.memory.rssMB‚ 2693.53125 disk.sda1.out‚0$ disk./.usageGB‚225.5589828491211 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.0546875 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 7569.02734375 disk.drbd2265.in‚0 disk.drbd2265.out‚0 cpu‚25.373206270785897 disk.rbd0.out‚0 disk.dm-0.in‚ 346.6796875 disk.sda.out‚ 7569.02734375 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 346.6796875 disk.sda.in‚ 0.0546875 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0~´zò2Ý- _wandbruntime‚4035  _runtime‚4035³y_Óé&ÞãM M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7035501731704 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101174860 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚141298408042360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167170644 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 trainer/global_step‚4999 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7055840836096 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67012309 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38538581 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28473217 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7178233311232 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 101154796 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚511 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67011798 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167190710 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148196605904384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚4035.111566149 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚156723415804928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66015848 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66015850 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚156723175002112 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 % _timestamp‚1.7883994104741685e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66015850 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚141140783071232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28473726 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7056009849344 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7056009849344 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _step‚410 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚141298590203332 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149544965630976 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149544941690880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚134262906310656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7178450173952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7055840836096 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167190710 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚148196792920576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚141140765068288 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚148196792920576 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66015848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚141140765068288 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7035666853316 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67011798 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 167190710 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚148196605904384 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 _runtime‚4035.111566149 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚15îÿ†P/6723415804928 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66015848 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚66015850 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚156723175002112 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 % _timestamp‚1.7883994104741685e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66015850 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚141140783071232 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28473726 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7056009849344 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7056009849344 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _step‚410 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚141298590203332 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚149544965630976 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚149544941690880 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚134262906310656 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7178450173952 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7055840836096 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 167190710 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚148196792920576 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚141140765068288 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚148196792920576 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66015848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚141140765068288 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7035666853316 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7035501731704 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 101174860 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚141298408042360 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 167170644 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 trainer/global_step‚4999š:"‡þ2à- _wandbruntime‚4035  _runtime‚4035ç±`VáÐ 0loss/FRt3/assignment_loss‚2.6062668207529427 /loss/FRt4/assignment_loss‚1.403202427517284 2loss/SRqqt3/assignment_loss‚1.7886144696766961 /loss/FBt1/detection_loss‚2.3719014687971636 _step‚411 0loss/FBt1/assignment_loss‚0.6983156948117539 /loss/FRt4/detection_loss‚1.6743045483123173 /loss/SRqqt2/detection_loss‚2.33600151055568 %loss/total_loss‚44.81365790964658 1loss/SRqqt4/assignment_loss‚1.396912332598327 .loss/FRt3/detection_loss‚2.336464711540454 1loss/SRqqt4/detection_loss‚1.1058630285681121 0loss/FRt1/assignment_loss‚3.2219231526056924 0loss/FRt2/assignment_loss‚3.8304453964891105 .loss/FRt2/detection_loss‚2.960905405981787 .loss/FBt2/detection_loss‚3.452029400401645 epoch‚4 .loss/FBt3/detection_loss‚2.590578520298004 trainer/global_step‚4999 _runtime‚ 4035.11297133 1loss/FBt2/assignment_loss‚0.36395548822122237 1loss/SRqqt1/detection_loss‚2.0542891241077865 2loss/SRqqt2/assignment_loss‚1.9326346826714438 .loss/FRt1/detection_loss‚2.131847103436788 /loss/FBt4/detection_loss‚0.8790930167479443 2loss/SRqqt1/assignment_loss‚1.4597211332459534 /loss/FBt3/assignment_loss‚0.470867723505944 1loss/FBt4/assignment_loss‚0.17587116092877172 1loss/SRqqt3/detection_loss‚1.5716495878737549 % _timestamp‚1.7883994104759226e+09†v[âÕ 0loss/FRt1/assignment_loss‚3.2219231526056924 0loss/FRt2/assignment_loss‚3.8304453964891105 .loss/FRt2/detection_loss‚2.960905405981787 .loss/FBt2/detection_loss‚3.452029400401645 epoch‚4 .loss/FBt3/detection_loss‚2.590578520298004 trainer/global_step‚4999 _runtime‚ 4035.11297133 1loss/FBt2/assignment_loss‚0.36395548822122237 1loss/SRqqt1/detection_loss‚2.0542891241077865 2loss/SRqqt2/assignment_loss‚1.9326346826714438 .loss/FRt1/detection_loss‚2.131847103436788 /loss/FBt4/detection_loss‚0.8790930167479443 2loss/SRqqt1/assignment_loss‚1.4597211332459534 /loss/FBt3/assignment_loss‚0.470867723505944 1loss/FBt4/assignment_loss‚0.17587116092877172 1loss/SRqqt3/detection_loss‚1.5716495878737549 % _timestamp‚1.7883994104759226e+09 0loss/FRt3/assignment_loss‚2.6062668207529427 /loss/FRt4/assignment_loss‚1.403202427517284 2loss/SRqqt3/assignment_loss‚1.7886144696766961 /loss/FBt1/detection_loss‚2.3719014687971636 _step‚411 0loss/FBt1/assignment_loss‚0.6983156948117539 /loss/FRt4/detection_loss‚1.6743045483123173 /loss/SRqqt2/detection_loss‚2.33600151055568 %loss/total_loss‚44.81365790964658 1loss/SRqqt4/assignment_loss‚1.396912332598327 .loss/FRt3/detection_loss‚2.336464711540454 1loss/SRqqt4/detection_loss‚1.1058630285681121›o7sã‚(:’ ÁžãÔÎÿ­ gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚131.312' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.262400000000003 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_ð[kqä‚(:æ ÁžãÔµ¼Ö° disk.sda2.out‚0 disk.dm-1.out‚ 7569.51171875 disk.rbd0.out‚0 disk.sda.in‚ 0.0546875 disk.drbd1716.in‚0 disk.rbd0.in‚0! disk.nvme0n1.out‚ 23798.921875 proc.cpu.threads‚17 disk.sda2.in‚0 disk.dm-0.out‚ 23798.921875 disk.rbd1.out‚13.03564453125! disk.nvme0c0n1.in‚ 346.6796875 disk.nvme0n1.in‚ 346.6796875* disk./.usagePercent‚15.137453875634572 disk.sda.out‚ 7569.51171875 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 346.6796875$ memory_percent‚93.75128746032715 proc.memory.rssMB‚ 2693.53125 network.sent‚ 1.1762604e+07 cpu‚25.371964968852648 disk.dm-1.in‚ 0.0546875 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.sda3.out‚ 7569.51171875 disk.rbd1.in‚0 disk.drbd2266.out‚0 disk.sda1.out‚0# disk.nvme0c0n1.out‚ 23798.921875) proc.memory.percent‚16.44001007080078 network.recv‚ 1.491064e+06 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.0546875( proc.memory.availableMB‚ 1023.7890625% disk./.usageGB‚225.55900192260742è!å‚(:’ ОãÔáÔ® gpu.0.gpu‚22.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚120.558' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.111600000000003 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âw%×yæ‚(:î ОãÔ¾åæ° disk.drbd2266.out‚0 disk.rbd0.out‚0) proc.memory.percent‚16.44001007080078 network.sent‚ 1.1860393e+07* disk./.usagePercent‚15.137452595596901 disk.sda.out‚ 7570.23046875 disk.dm-1.out‚ 7570.23046875! disk.dm-0.out‚23799.291015625$ memory_percent‚93.75078678131104$ disk./.usageGB‚225.5589828491211 disk.sda1.out‚0 disk.sda.in‚ 0.0546875 disk.drbd1716.in‚0$ disk.nvme0n1.out‚23799.291015625) proc.memory.availableMB‚ 1023.87109375! disk.nvme0c0n1.in‚ 346.6796875 disk.sda2.out‚0 disk.drbd2265.in‚0 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 network.recv‚ 1.496755e+06 disk.sda1.in‚0 disk.sda3.in‚ 0.0546875 disk.rbd1.out‚13.03564453125& disk.nvme0c0n1.out‚23799.291015625 disk.dm-1.in‚ 0.0546875 disk.drbd2080.out‚0 disk.dm-0.in‚ 346.6796875 disk.sda3.out‚ 7570.23046875 disk.rbd1.in‚0 disk.drbd2080.in‚0 proc.memory.rssMB‚ 2693.53125 cpu‚25.37084362167684 disk.nvme0n1.in‚ 346.6796875 disk.sda2.in‚0î˜É2ç- _wandbruntime‚4070  _runtime‚4070ÆÕEàÉèà _step‚412 ' lr-AdamW/pg1‚0.00015541700185002125 ' lr-AdamW/pg2‚0.00015541700185002125 trainer/global_step‚5049 % _timestamp‚1.7883994458973415e+09 _runtime‚4070.534223841òøVõÎéÈ _runtime‚4070.534223841 _step‚412 ' lr-AdamW/pg1‚0.00015541700185002125 ' lr-AdamW/pg2‚0.00015541700185002125 trainer/global_step‚5049 % _timestamp‚1.7883994458973415e+09œlòÔ·2ê- _wandbruntime‚4070  _runtime‚4070eäF>ß'ëÙO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28739584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 168763850 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚142637888791452 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7124288020480 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚149601082960896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66636139 trainer/global_step‚5049 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7124288020480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102127711 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚510 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7247889876480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7124119020544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168783890 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102127711 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7103584385568 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚149601082960896 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7103749506972 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67640148 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7247677194752 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚135534139284480 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66636141 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚142476981943296 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158209353348608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚150961700093952 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 % _timestamp‚1.7883994458981552e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38900056 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064Ùt7ûŽìÞO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚142476981943296 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚158209353348608 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚150961700093952 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 % _timestamp‚1.7883994458981552e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚38900056 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚28739584 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 168763850 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚142637888791452 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7124288020480 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚149601082960896 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚66636139 trainer/global_step‚5049 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7124288020480 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 102127711 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚510 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7247889876480 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7124119020544 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 168783890 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 102127711 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7103584385568 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚149601082960896 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 710374]l&V9506972 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚67640148 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7247677194752 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚135534139284480 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚66636141G8Þ¿2í- _wandbruntime‚4071  _runtime‚4071ë»gç&îáM VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38907431 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚135560082466816 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚135560065427456 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚142504233305088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚142504233305088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚150990613042176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚142504251308032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚150990589102080 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚142665038977568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168796296 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168816362 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142504251308032 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚149629745463808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7249307176960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102167562 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28744845 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7105138671724 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚5049 _runtime‚ 4071.20944939 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66648798 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 247094272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66648798 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7249084022784 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38907433 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 168796296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚158239920219136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚149629932480000 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚149629932480000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚142665221138540 $ _timestamp‚1.788399446572034e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7125512158720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚158239673124864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67652812 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7104973550112 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67652276 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223154176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66648800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7125512158720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28745379 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7125681171968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257g½j[ì&ïæM aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7249307176960 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 102167562 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚28744845 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7105138671724 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚5049 _runtime‚ 4071.20944939 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚66648798 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 247094272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚66648798 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7249084022784 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚38907433 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 168796296 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚158239920219136 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚414 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚149629932480000 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚149629932480000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚142665221138540 $ _timestamp‚1.788399446572034e+09 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7125512158720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚158239673124864 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚67652812 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7104973550112 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚67652276 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223154176 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚66648800 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7125512158720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚28745379 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7125681171968 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚38907431 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚135560082466816 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚135560065427456 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚142504233305088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚142504233305088 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚150990613042176 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚142504251308032 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚150990589102080 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚142665038977568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 168796296 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 168816362 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚142504251308032 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚149629745463808 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401ž¸-Y2ð- _wandbruntime‚4071  _runtime‚4071ñ8¡dYñÓ 0loss/FRt1/assignment_loss‚3.7013528138872176 /loss/FRt2/detection_loss‚3.2842446851730345 1loss/SRqqt1/detection_loss‚1.6734677709284282 .loss/FBt1/detection_loss‚3.043366849422455 1loss/FBt4/assignment_loss‚0.15994321238597073 _step‚415 2loss/SRqqt2/assignment_loss‚1.8942800068429537 1loss/FBt1/assignment_loss‚0.49531106240647205 /loss/FRt1/detection_loss‚1.8139521091703439 /loss/SRqqt3/detection_loss‚1.68433328717947 1loss/SRqqt4/detection_loss‚1.3489895619893983 /loss/FBt4/detection_loss‚0.8249358797410354 % _timestamp‚1.7883994465738149e+09 0loss/FRt4/assignment_loss‚1.8101501234553077 2loss/SRqqt1/assignment_loss‚1.6216141434741163 2loss/SRqqt3/assignment_loss‚1.5959951390201847 /loss/FRt4/detection_loss‚2.1228933104059915 %loss/total_loss‚48.33603631523763 trainer/global_step‚5049 .loss/FRt2/assignment_loss‚4.48787745475769 0loss/FBt3/assignment_loss‚0.7231284454464912 0loss/FBt2/assignment_loss‚0.2753665024737675 .loss/FRt3/detection_loss‚2.475427268367065 0loss/SRqqt2/detection_loss‚2.828253924846649 /loss/FBt2/detection_loss‚2.5303994665543237 2loss/SRqqt4/assignment_loss‚0.9856872898289712 /loss/FRt3/assignment_loss‚3.469168863798443 /loss/FBt3/detection_loss‚3.4858971436818442 epoch‚4 _runtime‚4071.210550077›ZI^òØ 1loss/FBt4/assignment_loss‚0.15994321238597073 _step‚415 2loss/SRqqt2/assignment_loss‚1.8942800068429537 1loss/FBt1/assignment_loss‚0.49531106240647205 /loss/FRt1/detection_loss‚1.8139521091703439 /loss/SRqqt3/detection_loss‚1.68433328717947 1loss/SRqqt4/detection_loss‚1.3489895619893983 /loss/FBt4/detection_loss‚0.8249358797410354 % _timestamp‚1.7883994465738149e+09 0loss/FRt4/assignment_loss‚1.8101501234553077 2loss/SRqqt1/assignment_loss‚1.6216141434741163 2loss/SRqqt3/assignment_loss‚1.5959951390201847 /loss/FRt4/detection_loss‚2.1228933104059915 %loss/total_loss‚48.33603631523763 trainer/global_step‚5049 .loss/FRt2/assignment_loss‚4.48787745475769 0loss/FBt3/assignment_loss‚0.7231284454464912 0loss/FBt2/assignment_loss‚0.2753665024737675 .loss/FRt3/detection_loss‚2.475427268367065 0loss/SRqqt2/detection_loss‚2.828253924846649 /loss/FBt2/detection_loss‚2.5303994665543237 2loss/SRqqt4/assignment_loss‚0.9856872898289712 /loss/FRt3/assignment_loss‚3.469168863798443 /loss/FBt3/detection_loss‚3.4858971436818442 epoch‚4 _runtime‚4071.210550077 0loss/FRt1/assignment_loss‚3.7013528138872176 /loss/FRt2/detection_loss‚3.2842446851730345 1loss/SRqqt1/detection_loss‚1.6734677709284282 .loss/FBt1/detection_loss‚3.043366849422455Ÿãï´ª’ó‚(:‡ ßžãԶͮ gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚144.824' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.9648 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U£!uô‚(:ê ßžãÔ›üû± disk.rbd1.in‚0 disk.drbd1716.in‚0 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.sda.out‚ 7570.609375 disk.drbd1716.out‚0$ disk./.usageGB‚225.5589828491211 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 346.6796875 disk.sda.in‚ 0.0546875) proc.memory.percent‚16.44001007080078 disk.drbd2266.in‚0 disk.drbd2266.out‚0 proc.memory.rssMB‚ 2693.53125 network.sent‚ 1.1866699e+07! disk.nvme0c0n1.in‚ 346.6796875$ memory_percent‚93.74933242797852* disk./.usagePercent‚15.137452595596901 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.out‚ 7570.609375 disk.drbd2080.out‚0 disk.rbd1.out‚13.18408203125 disk.sda2.out‚0 disk.dm-1.in‚ 0.0546875' proc.memory.availableMB‚ 1024.109375 network.recv‚ 1.501374e+06" disk.dm-0.out‚23800.1396484375 disk.sda1.out‚0' disk.nvme0c0n1.out‚23800.1396484375% disk.nvme0n1.out‚23800.1396484375 cpu‚25.369656331416984 disk.drbd2265.in‚0 disk.dm-0.in‚ 346.6796875 disk.sda3.in‚ 0.0546875 disk.sda3.out‚ 7570.609375#8ÿ‘õ‚(:† îžãԣ𒮠gpu.0.gpu‚28.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚121.625' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.325 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0µŸBÐ~ö‚(:ó îžãÔÃÔ“± disk.rbd0.out‚0 disk.dm-0.in‚ 346.6796875) proc.memory.percent‚16.44001007080078 proc.cpu.threads‚17" disk.dm-0.out‚23800.3818359375 network.recv‚ 1.506005e+06% disk./.usageGB‚225.55902481079102 network.sent‚ 1.1973466e+07 disk.drbd2265.out‚0 disk.rbd0.in‚0 proc.memory.rssMB‚ 2693.53125 disk.drbd2266.out‚0 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.0546875 disk.sda3.out‚ 7571.31640625 disk.rbd1.out‚13.18408203125' disk.nvme0c0n1.out‚23800.3818359375 disk.drbd2265.in‚0 disk.sda3.in‚ 0.0546875 disk.rbd1.in‚0% disk.nvme0n1.out‚23800.3818359375 disk.drbd2266.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 7571.31640625 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.nvme0n1.in‚ 346.6796875 disk.sda.out‚ 7571.31640625 disk.sda2.in‚0 disk.sda1.in‚0$ memory_percent‚93.75011920928955) proc.memory.availableMB‚ 1023.98046875* disk./.usagePercent‚15.137455411679776 disk.sda1.out‚0! disk.nvme0c0n1.in‚ 346.6796875 disk.sda.in‚ 0.0546875 cpu‚25.368442186442024ü+ 2÷- _wandbruntime‚4106  _runtime‚4106[™ ÛÉøà _step‚416 ' lr-AdamW/pg1‚0.00015402302299845732 ' lr-AdamW/pg2‚0.00015402302299845732 trainer/global_step‚5099 % _timestamp‚1.7883994818060343e+09 _runtime‚4106.442881305§3eÔÎùÈ trainer/global_step‚5099 % _timestamp‚1.7883994818060343e+09 _runtime‚4106.442881305 _step‚416 ' lr-AdamW/pg1‚0.00015402302299845732 ' lr-AdamW/pg2‚0.00015402302299845732 ™gr2ú- _wandbruntime‚4106  _runtime‚4106jŠ’¾;ûÙO ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67269091 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _runtime‚4106.443783908 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 170389502 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7318526497280 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚144004337565896 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7173221325380 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103120413 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚143840450180096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚151034222520320 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7193790343168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39268906 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7193959343104 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚152407347505152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚151034222520320 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚159726084587008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚5099 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7193790343168 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67269089 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚561 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7193959343104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor{º¤.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚151034409523200 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67269089 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29013506 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚143840432177152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚136831298401280 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67269091 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68282973 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7173056203976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103120413 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7883994818067358e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚143840432177152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7318737081856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚417 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚144004337565896 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7173221325380 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 103120413 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚143840450180096 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚151034222520320 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7193790343168 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚39268906 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7193959343104 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚152407347505152 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚151034222520320 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚159726084587008 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚5099 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7193790343168 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67269089 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚561 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7193959343104 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚151034409523200 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67269089 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29013506 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚143840432177152 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚136831298401280 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚67269091 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68282973 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7173056203976 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 103120413 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7883994818067358e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚143840432177152 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7318737081856 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚417 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39276283 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚151063072039424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68295742 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 _step‚418 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚143867701541888 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170421948 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67281748 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 % _timestamp‚1.7883994824895463e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 170421948 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7195352494592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚143867719544832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170442014 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159756415851520 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67281750 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103140200 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29019459 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7195352494592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 _runtime‚4107.126997596 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚144031852073748 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚144031669912776 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7174445368520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68295205 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚159756179243008 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚5099 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67281750 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7195183481344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7319942729728 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7174610490132 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67281748 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚151063072039424 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7320155398144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚143867701541888 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚136857224544256Eïô::ÿèM E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚144031669912776 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7174445368520 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68295205 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚159756179243008 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚5099 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67281750 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7195183481344 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7319942729728 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7174610490132 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚67281748 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚151063072039424 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7320155398144 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚143867701541888 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚136857224544256 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39276283 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚151063072039424 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68295742 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 _step‚418 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚143867701541888 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 170421948 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚67281748 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 % _timestamp‚1.7883994824895463e+09 D6DeviceStatsMonitor.on_train_batchßßá´ _end/active.all.freed‚ 170421948 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7195352494592 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚143867719544832 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 170442014 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚159756415851520 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚67281750 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 103140200 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29019459 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7195352494592 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 _runtime‚4107.126997596 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚144031852073748 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0¢tÉÿ2€- _wandbruntime‚4107  _runtime‚4107 R ¤WÑ /loss/FRt4/assignment_loss‚1.147108145058155 1loss/FBt4/assignment_loss‚0.18893987315443458 trainer/global_step‚5099 1loss/SRqqt1/assignment_loss‚2.009547011592449 0loss/FBt3/assignment_loss‚0.5937133704622587 .loss/FRt4/detection_loss‚1.702834893328448 /loss/FRt1/detection_loss‚2.3733973031820255 1loss/SRqqt1/detection_loss‚2.1295445332160363 1loss/SRqqt3/detection_loss‚1.5795422725139125 .loss/FBt2/detection_loss‚2.676614811643958 /loss/FBt3/detection_loss‚3.7482292453447976 1loss/SRqqt2/assignment_loss‚2.247907817363739 2loss/SRqqt3/assignment_loss‚1.5482397799890848 2loss/SRqqt4/assignment_loss‚1.1071259357739944 0loss/SRqqt2/detection_loss‚2.415327712893486 /loss/FBt1/detection_loss‚2.2344386738080244 .loss/FBt4/detection_loss‚0.570747050420864 /loss/FRt1/assignment_loss‚3.022795259952545 /loss/FRt2/detection_loss‚2.9473325472611647 1loss/SRqqt4/detection_loss‚1.0195457214439236 %loss/total_loss‚44.79615163533714 _runtime‚4107.128030666 0loss/FBt1/assignment_loss‚0.2504846227685938 _step‚419 .loss/FRt3/assignment_loss‚2.85994033842552 1loss/FBt2/assignment_loss‚0.33868227909805526 /loss/FRt3/detection_loss‚2.4018107594513314 epoch‚4 % _timestamp‚1.7883994824913003e+09 0loss/FRt2/assignment_loss‚3.6823016771903405³©n¾\‚Ö .loss/FRt3/assignment_loss‚2.85994033842552 1loss/FBt2/assignment_loss‚0.33868227909805526 /loss/FRt3/detection_loss‚2.4018107594513314 epoch‚4 % _timestamp‚1.7883994824913003e+09 0loss/FRt2/assignment_loss‚3.6823016771903405 /loss/FRt4/assignment_loss‚1.147108145058155 1loss/FBt4/assignment_loss‚0.18893987315443458 trainer/global_step‚5099 1loss/SRqqt1/assignment_loss‚2.009547011592449 0loss/FBt3/assignment_loss‚0.5937133704622587 .loss/FRt4/detection_loss‚1.702834893328448 /loss/FRt1/detection_loss‚2.3733973031820255 1loss/SRqqt1/detection_loss‚2.1295445332160363 1loss/SRqqt3/detection_loss‚1.5795422725139125 .loss/FBt2/detection_loss‚2.676614811643958 /loss/FBt3/detection_loss‚3.7482292453447976 1loss/SRqqt2/assignment_loss‚2.247907817363739 2loss/SRqqt3/assignment_loss‚1.5482397799890848 2loss/SRqqt4/assignment_loss‚1.1071259357739944 0loss/SRqqt2/detection_loss‚2.415327712893486 /loss/FBt1/detection_loss‚2.2344386738080244 .loss/FBt4/detection_loss‚0.570747050420864 /loss/FRt1/assignment_loss‚3.022795259952545 /loss/FRt2/detection_loss‚2.9473325472611647 1loss/SRqqt4/detection_loss‚1.0195457214439236 %loss/total_loss‚44.79615163533714 _runtime‚4107.128030666 0loss/FBt1/assignment_loss‚0.2504846227685938 _step‚419£êÁ ’ƒ‚(:‡ ýžãÔÚìž® gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚120.262' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0524 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)KÆŸl„‚(:á ýžãÔ–¡á± disk.dm-1.out‚ 7571.6875 disk.rbd0.out‚0 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.0546875 disk.drbd2266.in‚0 disk.rbd1.out‚13.21630859375! disk.nvme0c0n1.in‚ 346.6796875$ memory_percent‚93.75033378601074 disk.rbd0.in‚0) proc.memory.percent‚16.44001007080078 disk.dm-0.in‚ 346.6796875 disk.drbd1716.out‚0( proc.memory.availableMB‚ 1023.9453125 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 7571.6875 disk.drbd2080.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 network.recv‚ 1.510379e+06! disk.dm-0.out‚23800.861328125 disk.nvme0n1.in‚ 346.6796875$ disk.nvme0n1.out‚23800.861328125 disk.sda2.in‚0 disk.sda2.out‚0% disk./.usageGB‚225.55902862548828 disk.sda1.in‚0 disk.sda3.in‚ 0.0546875& disk.nvme0c0n1.out‚23800.861328125 disk.drbd2080.in‚0 proc.memory.rssMB‚ 2693.53125 cpu‚25.36733990059624 network.sent‚ 1.1979357e+07) disk./.usagePercent‚15.13745566768731 disk.sda1.out‚0 disk.sda3.out‚ 7571.6875 disk.sda.in‚ 0.0546875ð"@9’…‚(:‡ ŒŸãÔì¿®® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚123.751' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7502 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0éÝÝ*s†‚(:è ŒŸãÔô¬ ±$ disk./.usageGB‚225.5589485168457 disk.rbd0.in‚0 cpu‚25.366199130854966 disk.sda1.out‚0 disk.sda1.in‚0 network.recv‚ 1.516623e+06 disk.drbd2080.out‚0 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 346.6796875 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda3.out‚ 7572.3984375 disk.dm-1.in‚ 0.0546875$ memory_percent‚93.74885559082031! disk.dm-0.out‚23801.138671875 disk.nvme0n1.in‚ 346.6796875 disk.sda.in‚ 0.0546875 disk.sda2.in‚0 disk.drbd1716.in‚0) proc.memory.percent‚16.44001007080078* disk./.usagePercent‚15.137450291529092 disk.dm-0.in‚ 346.6796875 disk.rbd1.in‚0$ disk.nvme0n1.out‚23801.138671875 disk.sda.out‚ 7572.3984375 disk.sda2.out‚0 disk.dm-1.out‚ 7572.3984375 disk.drbd2266.out‚0 network.sent‚ 1.2087984e+07 disk.drbd2080.in‚0% proc.memory.availableMB‚ 1024.1875 proc.memory.rssMB‚ 2693.53125 proc.cpu.threads‚17 disk.sda3.in‚ 0.0546875 disk.rbd1.out‚13.21630859375& disk.nvme0c0n1.out‚23801.138671875 disk.drbd1716.out‚0?Øð“’‡‚(:‡ ›ŸãÔèÖ® gpu.0.gpu‚14.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚112.708' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5416 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏÎýkˆ‚(:à ›ŸãÔ÷‚ô°) proc.memory.percent‚16.44001007080078* disk./.usagePercent‚15.137450291529092 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.sda3.in‚ 0.0546875 disk.dm-1.out‚ 7572.84375 cpu‚25.365004725908157% disk.nvme0c0n1.out‚23801.66015625# disk.nvme0n1.out‚23801.66015625 disk.sda.out‚ 7572.84375 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0$ disk./.usageGB‚225.5589485168457 disk.sda3.out‚ 7572.84375 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 network.sent‚ 1.2094247e+07 disk.sda1.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.0546875( proc.memory.availableMB‚ 1023.4765625 proc.cpu.threads‚17 network.recv‚ 1.521171e+06 disk.dm-0.in‚ 346.6796875! disk.nvme0c0n1.in‚ 346.6796875 disk.dm-1.in‚ 0.0546875 disk.drbd2266.in‚0 disk.nvme0n1.in‚ 346.6796875 proc.memory.rssMB‚ 2693.53125 disk.dm-0.out‚23801.66015625 disk.rbd1.out‚ 13.279296875 disk.drbd2266.out‚0$ memory_percent‚93.75319480895996VnÝü2‰- _wandbruntime‚4142  _runtime‚4142É%é"ÉŠÃ ' lr-AdamW/pg1‚0.00015261934742378215 ' lr-AdamW/pg2‚0.00015261934742378215 trainer/global_step‚5149 % _timestamp‚1.7883995181313608e+09 _runtime‚4142.768232137 _step‚420 lÎèÎ‹È ' lr-AdamW/pg1‚0.00015261934742378215 ' lr-AdamW/pg2‚0.00015261934742378215 trainer/global_step‚5149 % _timestamp‚1.7883995181313608e+09 _runtime‚4142.768232137 _step‚420¤Ê÷v#2Œ- _wandbruntime‚4142  _runtime‚4142ÌC¬Êß'ÙO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7242693143788 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚145370968501104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104113115 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7389405874176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7263630665728 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _step‚421 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67902039 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚138128440478720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67902039 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7883995181320527e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29285177 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚145203918416896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67902041 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚152467549082624 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68923475 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104133153 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 172035194 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚153852970976256 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚138128457518080 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161242376850432 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚153852994916352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚145371150661868 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚145203900413952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚5149 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚145203900413952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104133153 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7389620653056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104113115 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39637754 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29285719 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161242615569408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7263461665792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7263630665728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 172015154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145203918416896 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152467362079744Ò6†ä'ŽÞO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚67902039 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚138128440478720 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚67902039 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7883995181320527e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29285177 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚145203918416896 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚67902041 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚152467549082624 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚68923475 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 104133153 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 172035194 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚153852970976256 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚138128457518080 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚161242376850432 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚153852994916352 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚145371150661868 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚145203900413952 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚5149 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚145203900413952 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 104133153 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7389620653056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 104113115 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚39637754 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29285719 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚161242615569408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7263461665792 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7263630665728 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 172015154 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚145203918416896 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚152467362079744 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7242693143788 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚145370968501104 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 104113115 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7389405874176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7263630665728 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _step‚421¥n‹{2- _wandbruntime‚4143  _runtime‚4143n‚·ßùãM UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7265023817216 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚153881883924480 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 % _timestamp‚1.7883995188098695e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152496024582656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚145231169778688 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172047600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚145231187781632 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104152966 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39645133 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚138154400700416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 172047600 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7391037606400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67914700 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7243917186928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚152496211598848 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39645131 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68936358 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚145398300847984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚145231187781632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7265023817216 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68935830 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7390816549376 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152496211598848 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104132902 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161272945470976 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104152966 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚145231169778688 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚422 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29291225 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29290699 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172067666 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 172067666 _runtime‚4143.447213182 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161272700473856 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚153881907864576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 221057024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7264854803968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244997120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚145398483008956 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7244082308540 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0"Õ¢î&‘èM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚67914700 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7243917186928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚152496211598848 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚39645131 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚68936358 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚145398300847984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚145231187781632 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7265023817216 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚68935830 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7390816549376 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚152496211598848 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 104132902 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚161272945470976 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 104152966 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚145231169778688 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚422 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29291225 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29290699 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 172067666 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 172067666 _runtime‚4143.447213182 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚161272700473856 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚153881907864576 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 221057024 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7264854803968 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244997120 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚145398483008956 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7244082308540 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7265023817216 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚153881883924480 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 % _timestamp‚1.7883995188098695e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚152496024582656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚145231169778688 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 172047600 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚145231187781632 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 104152966 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚39645133 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚138154400700416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 172047600 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7391037606400 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416¦‘£|2’- _wandbruntime‚4143  _runtime‚4143и{X“Ò .loss/FBt3/detection_loss‚5.067356586456299 0loss/FRt4/assignment_loss‚1.9517142001320333 2loss/SRqqt1/assignment_loss‚2.0284501630812883 1loss/SRqqt2/assignment_loss‚1.891031535807997 _runtime‚4143.448262723 2loss/SRqqt3/assignment_loss‚1.4216643007472158 1loss/SRqqt1/detection_loss‚1.8284213918128185 1loss/SRqqt4/detection_loss‚1.1075541441855223 /loss/FBt4/detection_loss‚0.6779591228059005 %loss/total_loss‚48.91446031278197 epoch‚4 trainer/global_step‚5149 1loss/FBt3/assignment_loss‚0.11336930762869694 /loss/FBt2/detection_loss‚3.7872968435287477 /loss/FBt1/detection_loss‚2.5958282080563633 .loss/FRt1/detection_loss‚2.030623768505297 /loss/FRt2/assignment_loss‚4.331250031789144 /loss/FRt3/assignment_loss‚2.661770986749771 0loss/SRqqt2/detection_loss‚2.651263468898833 1loss/SRqqt3/detection_loss‚2.1254214350134135 _step‚423 /loss/FRt1/assignment_loss‚3.238737549698144 2loss/SRqqt4/assignment_loss‚1.0208724134642144 .loss/FRt3/detection_loss‚2.513359366579259 $ _timestamp‚1.788399518811511e+09 1loss/FBt1/assignment_loss‚0.18246484124442344 0loss/FBt2/assignment_loss‚0.6291745334863663 0loss/FBt4/assignment_loss‚0.1529739877306015 /loss/FRt4/detection_loss‚1.9949348449122672 /loss/FRt2/detection_loss‚2.9109672804673514nUÎŒ]”× _runtime‚4143.448262723 2loss/SRqqt3/assignment_loss‚1.4216643007472158 1loss/SRqqt1/detection_loss‚1.8284213918128185 1loss/SRqqt4/detection_loss‚1.1075541441855223 /loss/FBt4/detection_loss‚0.6779591228059005 %loss/total_loss‚48.91446031278197 epoch‚4 trainer/global_step‚5149 1loss/FBt3/assignment_loss‚0.11336930762869694 /loss/FBt2/detection_loss‚3.7872968435287477 /loss/FBt1/detection_loss‚2.5958282080563633 .loss/FRt1/detection_loss‚2.030623768505297 /loss/FRt2/assignment_loss‚4.331250031789144 /loss/FRt3/assignment_loss‚2.661770986749771 0loss/SRqqt2/detection_loss‚2.651263468898833 1loss/SRqqt3/detection_loss‚2.1254214350134135 _step‚423 /loss/FRt1/assignment_loss‚3.238737549698144 2loss/SRqqt4/assignment_loss‚1.0208724134642144 .loss/FRt3/detection_loss‚2.513359366579259 $ _timestamp‚1.788399518811511e+09 1loss/FBt1/assignment_loss‚0.18246484124442344 0loss/FBt2/assignment_loss‚0.6291745334863663 0loss/FBt4/assignment_loss‚0.1529739877306015 /loss/FRt4/detection_loss‚1.9949348449122672 /loss/FRt2/detection_loss‚2.9109672804673514 .loss/FBt3/detection_loss‚5.067356586456299 0loss/FRt4/assignment_loss‚1.9517142001320333 2loss/SRqqt1/assignment_loss‚2.0284501630812883 1loss/SRqqt2/assignment_loss‚1.891031535807997§`†Å’•‚(:‡ ªŸãÔ¯²û­ gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚135.378' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.0756 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a™òj–‚(:ß ªŸãÔÙ¥´±* disk./.usagePercent‚15.137454131642105 disk.sda.in‚ 0.0546875 disk.drbd2265.in‚0 disk.drbd2080.out‚0! disk.nvme0c0n1.in‚ 346.6796875 disk.drbd2080.in‚0 disk.dm-0.out‚ 23802.1015625 disk.sda3.out‚ 7573.546875 disk.drbd1716.out‚0$ memory_percent‚93.75200271606445 proc.cpu.threads‚17$ disk./.usageGB‚225.5590057373047 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.0546875' proc.memory.availableMB‚ 1023.671875 disk.rbd1.in‚0 proc.memory.rssMB‚ 2693.53125 network.recv‚ 1.524722e+06 disk.nvme0n1.in‚ 346.6796875 disk.dm-1.out‚ 7573.546875 disk.drbd2265.out‚0 disk.rbd0.out‚0) proc.memory.percent‚16.44001007080078 cpu‚25.363868138616358 disk.rbd1.out‚ 13.279296875$ disk.nvme0c0n1.out‚ 23802.1015625" disk.nvme0n1.out‚ 23802.1015625 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.0546875 disk.drbd2266.in‚0 disk.dm-0.in‚ 346.6796875 network.sent‚ 1.2099731e+07 disk.sda.out‚ 7573.546875 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.sda1.out‚0U¨¯ý—‚(:’ ¹ŸãÔ —™® gpu.0.gpu‚34.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚120.558' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.111600000000003 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à²] v˜‚(:ë ¹ŸãÔøõΰ% disk.nvme0n1.out‚23802.5576171875 proc.memory.rssMB‚ 2693.53125 proc.cpu.threads‚17 network.sent‚ 1.2208004e+07 disk.drbd2080.in‚0 disk.sda3.out‚ 7574.046875) proc.memory.percent‚16.44001007080078 disk.dm-1.in‚ 0.0546875 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚23802.5576171875 disk.sda2.out‚0$ memory_percent‚93.75050067901611 cpu‚25.362696074816967 disk.drbd1716.in‚0 disk.drbd1716.out‚0" disk.dm-0.out‚23802.5576171875 disk.rbd0.out‚0! disk.nvme0c0n1.in‚ 346.6796875 disk.sda2.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 346.6796875 disk.sda1.out‚0 disk.drbd2265.out‚0* disk./.usagePercent‚15.137448755483888$ disk./.usageGB‚225.5589256286621 disk.drbd2266.in‚0 disk.sda3.in‚ 0.0546875 disk.sda.in‚ 0.0546875 disk.sda.out‚ 7574.046875 disk.nvme0n1.in‚ 346.6796875) proc.memory.availableMB‚ 1023.91796875 network.recv‚ 1.53097e+06 disk.dm-1.out‚ 7574.046875 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.rbd1.out‚13.34228515625+*‘ƒ2™- _wandbruntime‚4178  _runtime‚4178²Õôqɚà ' lr-AdamW/pg1‚0.00015120626764988172 ' lr-AdamW/pg2‚0.00015120626764988172 trainer/global_step‚5199 % _timestamp‚1.7883995539417222e+09 _runtime‚4178.578539135 _step‚424œß+Î›È % _timestamp‚1.7883995539417222e+09 _runtime‚4178.578539135 _step‚424 ' lr-AdamW/pg1‚0.00015120626764988172 ' lr-AdamW/pg2‚0.00015120626764988172 trainer/global_step‚5199¨·³:\2œ- _wandbruntime‚4178  _runtime‚4178½E ÏÙO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚146737599436312 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68534989 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚155298618387456 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _runtime‚4178.579500359 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7333132988416 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7333301988352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7333132988416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40006606 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚162758952678400 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7333301988352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68534991 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7460551166976 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40006604 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68534989 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69564591 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105125855 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚146567368650752 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚162759193494528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7460334290944 trainer/global_step‚5199 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚146567368650752 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚155298642327552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚153900501639168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69565145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68534991 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105125855 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173640806 M?DeviceStatsMonitor.on_train_batch_start/active.smaÛLÓ ll_pool.freed‚ 105105817 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚146737781597076 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29558539 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚146567386653696 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚153900688642048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173660846 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29557987 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚139425599595520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7312164962196 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105105817 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚146567386653696 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚139425616634880 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚153900688642048 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7311999840792 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0ˈÈä'žÞO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 173640806 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 105105817 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚146737781597076 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29558539 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚146567386653696 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚153900688642048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 173660846 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29557987 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚139425599595520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7312164962196 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 105105817 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚146567386653696 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚139425616634880 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚153900688642048 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7311999840792 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚146737599436312 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚68534989 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚155298618387456 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _runtime‚4178.579500359 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7333132988416 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7333301988352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7333132988416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40006606 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚162758952678400 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7333301988352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚68534991 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7460551166976 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40006604 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚68534989 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚69564591 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 105125855 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚554 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚146567368650752 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚162759193494528 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7460334290944 trainer/global_step‚5199 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚146567368650752 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚155298642327552 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚153900501639168 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚69565145 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚68534991 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 105125855 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0©ÏÃ2Ÿ- _wandbruntime‚4179  _runtime‚41792ç,>è& âM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173673252 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚139451542777856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7313389005336 trainer/global_step‚5199 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7334695139840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105125604 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7334526126592 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚426 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚146594638015488 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68547650 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚146764931783192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69577350 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69577877 $ _timestamp‚1.788399554849562e+09 _runtime‚4179.486933158 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105145668 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚153929351158272 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7313554126948 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173693318 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173693318 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105125604 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68547648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68547648 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚153929351158272 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7461963519488 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚146594656018432 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29563894 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚162789518795264 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚146594656018432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29563369 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7334695139840 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40013983 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40013981 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7461748753920 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68547650 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105145668 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚139451559817216 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚146765113944164 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416M#:UÏ$¡çM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 _step‚426 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚146594638015488 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚68547650 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚146764931783192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚69577350 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚69577877 $ _timestamp‚1.788399554849562e+09 _runtime‚4179.486933158 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 105145668 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚153929351158272 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7313554126948 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 173693318 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 173693318 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 105125604 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚68547648 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚68547648 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚153929351158272 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7461963519488 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚146594656018432 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29563894 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚162789518795264 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚146594656018432 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚29563369 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7334695139840 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40013983 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40013981 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7461748753920 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚68547650 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 105145668 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚139451559817216 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚146765113944164 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 173673252 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚139451542777856 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7313389005336 trainer/global_step‚5199 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7334695139840 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 105125604 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7334526126592 F2¢- _wandbruntime‚4179  _runtime‚4179!g¤/[£Õ 1loss/FBt2/assignment_loss‚0.16483383677715224 0loss/FRt4/assignment_loss‚1.5527330935001373 1loss/FBt3/assignment_loss‚0.35838476044250456 1loss/SRqqt3/detection_loss‚2.3849002064930067 .loss/FBt3/detection_loss‚2.920775927603245 2loss/SRqqt4/assignment_loss‚1.1411091876083186 0loss/SRqqt1/detection_loss‚1.826178635592046 .loss/FBt1/detection_loss‚4.185384917259216 /loss/FBt4/detection_loss‚0.7007502754490367 /loss/FRt2/assignment_loss‚4.163525216281414 2loss/SRqqt3/assignment_loss‚1.5229760862679944 /loss/FRt2/detection_loss‚2.9020644035190344 1loss/SRqqt4/detection_loss‚1.0940838301820415 /loss/FBt2/detection_loss‚3.2562306438173567 _runtime‚4179.488002235 /loss/FRt1/detection_loss‚1.9666410684585571 % _timestamp‚1.7883995548512523e+09 trainer/global_step‚5199 /loss/FRt3/assignment_loss‚3.353066013409541 /loss/FRt4/detection_loss‚1.7066476051178243 1loss/SRqqt2/detection_loss‚2.4676515658696494 epoch‚4 0loss/FRt1/assignment_loss‚3.3547428055451465 1loss/SRqqt1/assignment_loss‚1.255176307916965 1loss/FBt1/assignment_loss‚0.09560164298745803 1loss/FBt4/assignment_loss‚0.18154090919023386 &loss/total_loss‚48.122562172529356 _step‚427 0loss/SRqqt2/assignment_loss‚2.19431353337837 .loss/FRt3/detection_loss‚3.373249699863104œãm`¤Ú /loss/FRt2/assignment_loss‚4.163525216281414 2loss/SRqqt3/assignment_loss‚1.5229760862679944 /loss/FRt2/detection_loss‚2.9020644035190344 1loss/SRqqt4/detection_loss‚1.0940838301820415 /loss/FBt2/detection_loss‚3.2562306438173567 _runtime‚4179.488002235 /loss/FRt1/detection_loss‚1.9666410684585571 % _timestamp‚1.7883995548512523e+09 trainer/global_step‚5199 /loss/FRt3/assignment_loss‚3.353066013409541 /loss/FRt4/detection_loss‚1.7066476051178243 1loss/SRqqt2/detection_loss‚2.4676515658696494 epoch‚4 0loss/FRt1/assignment_loss‚3.3547428055451465 1loss/SRqqt1/assignment_loss‚1.255176307916965 1loss/FBt1/assignment_loss‚0.09560164298745803 1loss/FBt4/assignment_loss‚0.18154090919023386 &loss/total_loss‚48.122562172529356 _step‚427 0loss/SRqqt2/assignment_loss‚2.19431353337837 .loss/FRt3/detection_loss‚3.373249699863104 1loss/FBt2/assignment_loss‚0.16483383677715224 0loss/FRt4/assignment_loss‚1.5527330935001373 1loss/FBt3/assignment_loss‚0.35838476044250456 1loss/SRqqt3/detection_loss‚2.3849002064930067 .loss/FBt3/detection_loss‚2.920775927603245 2loss/SRqqt4/assignment_loss‚1.1411091876083186 0loss/SRqqt1/detection_loss‚1.826178635592046 .loss/FBt1/detection_loss‚4.185384917259216 /loss/FBt4/detection_loss‚0.7007502754490367«!.È’¥‚(:‡ ÈŸãÔáå® gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚122.684' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.5368 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¾-Vé}¦‚(:ò ÈŸãÔû°$ memory_percent‚93.75219345092773) proc.memory.percent‚16.44001007080078 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0' proc.memory.availableMB‚ 1023.640625( disk.nvme0c0n1.out‚23802.99462890625& disk.nvme0n1.out‚23802.99462890625 disk.sda.in‚ 0.0546875 disk.sda.out‚ 7574.91015625$ disk./.usageGB‚225.5589485168457 disk.nvme0n1.in‚ 346.7578125 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.dm-0.in‚ 346.7578125# disk.dm-0.out‚23802.99462890625 disk.sda3.in‚ 0.0546875 disk.rbd1.in‚0 disk.rbd1.out‚13.34228515625* disk./.usagePercent‚15.137450291529092! disk.nvme0c0n1.in‚ 346.7578125 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.drbd1716.in‚0 disk.sda3.out‚ 7574.91015625 proc.cpu.threads‚17 network.sent‚ 1.2213926e+07 disk.dm-1.out‚ 7574.91015625 disk.drbd2266.in‚0 disk.rbd0.in‚0 proc.memory.rssMB‚ 2693.53125 cpu‚25.36158839961808 network.recv‚ 1.534807e+06 disk.dm-1.in‚ 0.0546875 disk.drbd2080.in‚0 disk.sda1.out‚0jRM ’§‚(:‡ ןãÔЯ¢® gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚127.528' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.5056 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Á¸–‚¨‚(:÷ ןãÔé€ý°( disk.nvme0c0n1.out‚23812.81689453125 network.recv‚ 1.541458e+06* disk./.usagePercent‚15.133719493732608 disk.drbd2080.in‚0 disk.dm-0.in‚ 350.02734375# disk.dm-0.out‚23812.81689453125 disk.nvme0n1.in‚ 350.02734375 disk.sda.out‚ 7575.96484375 network.sent‚ 1.2322602e+07% disk./.usageGB‚225.50335693359375 disk.drbd2080.out‚0 disk.rbd0.out‚0 cpu‚25.36047121418194 disk.dm-1.in‚ 0.0546875 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 350.02734375( proc.memory.availableMB‚ 1023.6015625 disk.sda2.out‚0 disk.dm-1.out‚ 7575.96484375 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 7575.96484375) proc.memory.percent‚16.44001007080078 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.rbd1.out‚13.34228515625 proc.memory.rssMB‚ 2693.53125 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.0546875& disk.nvme0n1.out‚23812.81689453125 disk.sda.in‚ 0.0546875$ memory_percent‚93.75243186950684 proc.cpu.threads‚17tˆù°2©- _wandbruntime‚4214  _runtime‚4214Ù`)ɪà _runtime‚4214.838236811 _step‚428 ' lr-AdamW/pg1‚0.00014978407816046288 ' lr-AdamW/pg2‚0.00014978407816046288 trainer/global_step‚5249 % _timestamp‚1.7883995902012691e+09Òß$,Î«È _runtime‚4214.838236811 _step‚428 ' lr-AdamW/pg1‚0.00014978407816046288 ' lr-AdamW/pg2‚0.00014978407816046288 trainer/global_step‚5249 % _timestamp‚1.7883995902012691e+09¬èRo2¬- _wandbruntime‚4214  _runtime‚4214Áp™ß'­ÙO cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7531437688320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚155333641198592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40375456 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7381471659200 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚164275501291008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚148104230371520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7381636780604 trainer/global_step‚5249 _step‚429 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚147930836887552 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚140722758712320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70207106 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7402804311040 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚156744265798656 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚155333828201472 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106118557 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7531235492352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚147930836887552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69167939 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29831650 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226136064 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202195968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 % _timestamp‚1.7883995902020931e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7402804311040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚147930854890496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7402973310976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175266458 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚155333641198592 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69167941 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106098519 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175286498 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29831128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164275727427072 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚147930854890496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚524 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40375454 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚4214.839164272 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106118557 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚140722775751680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚155333828201472 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7402973310976 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚148104412532284 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 43902269442ú²6ä'®ÞO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚155333828201472 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 106118557 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7531235492352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚147930836887552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69167939 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚29831650 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226136064 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202195968 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 % _timestamp‚1.7883995902020931e+09 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7402804311040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚147930854890496 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7402973310976 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 175266458 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚155333641198592 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69167941 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 106098519 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 175286498 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚29831128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚164275727427072 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚147930854890496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚524 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚40375454 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚4214.839164272 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 106118557 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚140722775751680 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚155333828201472 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7402973310976 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚148104412532284 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7531437688320 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚155333641198592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚40375456 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7381471659200 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚164275501291008 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚148104230371520 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7381636780604 trainer/global_step‚5249 _step‚429 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚147930836887552 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚140722758712320 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70207106 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7402804311040 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚156744265798656 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248­Ø¥Zþ‘¯‚(:† æŸãÔÿó®® gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚138.625' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.725 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ZlG€°‚(:õ æŸãÔŸÚ“± disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.0546875 disk.rbd1.in‚0' disk.nvme0c0n1.out‚23814.3232421875 proc.memory.rssMB‚ 2693.53125 disk.dm-1.in‚ 0.0546875 disk.sda1.out‚0 disk.sda3.out‚ 7576.52734375* disk./.usagePercent‚15.133780167518223 disk.sda.in‚ 0.0546875 disk.dm-1.out‚ 7576.52734375 disk.drbd2080.out‚0 disk.dm-0.in‚ 364.14453125 network.sent‚ 1.2327884e+07 disk.drbd2265.out‚0 disk.drbd2080.in‚0" disk.nvme0c0n1.in‚ 364.14453125 disk.drbd1716.out‚0 disk.drbd2266.in‚0) proc.memory.availableMB‚ 1023.60546875) proc.memory.percent‚16.44001007080078 cpu‚25.359429536340272 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.drbd2265.in‚0 network.recv‚ 1.544742e+06$ disk./.usageGB‚225.5042610168457 disk.sda2.in‚0 disk.sda2.out‚0 disk.rbd0.out‚0 disk.rbd1.out‚13.34228515625% disk.nvme0n1.out‚23814.3232421875$ memory_percent‚93.75240802764893 disk.drbd1716.in‚0" disk.dm-0.out‚23814.3232421875 disk.nvme0n1.in‚ 364.14453125 disk.sda.out‚ 7576.52734375Ç2Á›2±- _wandbruntime‚4215  _runtime‚4215ßÓ¦ïS²âM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106138370 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚550 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚147958124255232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29837335 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚155362490717696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69180600 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164305834059776 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40382833 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175318970 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70219618 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40382831 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚147958106252288 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚164306064376832 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106138370 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7404366462464 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7383025945356 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7532861689856 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7404366462464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚156773202686976 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175318970 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70220168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚548 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69180600 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69180598 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚147958106252288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚156773178746880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚148131744879372 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚140748701894656 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚147958124255232 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7404197449216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175298904 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 $ _timestamp‚1.788399590886965e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7532655312896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7404197449216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚148131562718400 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0áPUsí&³çM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚147958106252288 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚156773178746880 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚148131744879372 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚140748701894656 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚147958124255232 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7404197449216 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 175298904 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 $ _timestamp‚1.788399590886965e+09 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7532655312896 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7404197449216 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚148131562718400 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 106138370 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚550 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚147958124255232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚29837335 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚155362490717696 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚69180600 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚164305834059776 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40382833 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 175318970 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70219618 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40382831 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚147958106252288 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚164306064376832 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 106138370 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7404366462464 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7383025945356 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7532861689856 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7404366462464 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚156773202686976 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 175318970 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70220168 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚548 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69180600 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69180598®[«jD2´- _wandbruntime‚4215  _runtime‚4215bé[[µÕ /loss/FRt4/detection_loss‚1.8736557160382685 /loss/FBt4/detection_loss‚0.8559872296887483 /loss/FRt1/assignment_loss‚2.872025014186392 0loss/FBt2/assignment_loss‚0.5667756706476211 0loss/FBt3/assignment_loss‚0.5100417920039035 /loss/FRt1/detection_loss‚2.0447262008579408 /loss/FBt2/detection_loss‚3.5698838114738463 trainer/global_step‚5249 % _timestamp‚1.7883995908884542e+09 1loss/FBt1/assignment_loss‚0.23854603652179662 /loss/FRt3/detection_loss‚2.2776304384072623 0loss/FRt3/assignment_loss‚3.1290954252084098 2loss/SRqqt1/assignment_loss‚1.3097324420511722 2loss/SRqqt2/assignment_loss‚1.9091303845246632 2loss/SRqqt4/assignment_loss‚1.1607744343312723 /loss/FBt1/detection_loss‚1.9853962063789368 epoch‚4 _step‚431 /loss/FRt2/detection_loss‚2.8551341065993676 1loss/SRqqt1/detection_loss‚1.8627710270881652 .loss/FBt3/detection_loss‚3.096859077612559 %loss/total_loss‚45.74543776385029 2loss/SRqqt3/assignment_loss‚1.5592924798357075 1loss/FBt4/assignment_loss‚0.25139725500041266 1loss/SRqqt3/detection_loss‚1.7047368074799407 _runtime‚4215.525247488 .loss/FRt2/assignment_loss‚4.35756163413708 0loss/FRt4/assignment_loss‚1.8902602545593097 0loss/SRqqt2/detection_loss‚2.532087990731904 0loss/SRqqt4/detection_loss‚1.331936328485608,™`¶Ú /loss/FRt1/assignment_loss‚2.872025014186392 0loss/FBt2/assignment_loss‚0.5667756706476211 0loss/FBt3/assignment_loss‚0.5100417920039035 /loss/FRt1/detection_loss‚2.0447262008579408 /loss/FBt2/detection_loss‚3.5698838114738463 trainer/global_step‚5249 % _timestamp‚1.7883995908884542e+09 1loss/FBt1/assignment_loss‚0.23854603652179662 /loss/FRt3/detection_loss‚2.2776304384072623 0loss/FRt3/assignment_loss‚3.1290954252084098 2loss/SRqqt1/assignment_loss‚1.3097324420511722 2loss/SRqqt2/assignment_loss‚1.9091303845246632 2loss/SRqqt4/assignment_loss‚1.1607744343312723 /loss/FBt1/detection_loss‚1.9853962063789368 epoch‚4 _step‚431 /loss/FRt2/detection_loss‚2.8551341065993676 1loss/SRqqt1/detection_loss‚1.8627710270881652 .loss/FBt3/detection_loss‚3.096859077612559 %loss/total_loss‚45.74543776385029 2loss/SRqqt3/assignment_loss‚1.5592924798357075 1loss/FBt4/assignment_loss‚0.25139725500041266 1loss/SRqqt3/detection_loss‚1.7047368074799407 _runtime‚4215.525247488 .loss/FRt2/assignment_loss‚4.35756163413708 0loss/FRt4/assignment_loss‚1.8902602545593097 0loss/SRqqt2/detection_loss‚2.532087990731904 0loss/SRqqt4/detection_loss‚1.331936328485608 /loss/FRt4/detection_loss‚1.8736557160382685 /loss/FBt4/detection_loss‚0.8559872296887483¯và„ä·‚(:’ õŸãÔýž¹® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚121.088' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.217599999999997 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0B˜mg¸‚(:Ü õŸãÔª­“± disk.drbd2265.out‚0$ memory_percent‚93.75267028808594 network.recv‚ 1.550691e+06$ disk.nvme0c0n1.out‚ 23816.5234375 disk.nvme0n1.in‚ 364.609375 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 network.sent‚ 1.2425028e+07 disk.rbd1.in‚0 disk.sda.in‚ 0.0546875 disk.sda.out‚ 7577.046875 disk.sda2.in‚0 disk.drbd2265.in‚0 proc.memory.rssMB‚ 2693.53125 disk.sda1.in‚0 disk.sda3.out‚ 7577.046875" disk.nvme0n1.out‚ 23816.5234375 disk.rbd0.out‚0 cpu‚25.35827996519027 proc.cpu.threads‚17 disk.dm-0.in‚ 364.609375 disk.dm-0.out‚ 23816.5234375 disk.drbd2080.in‚0) proc.memory.percent‚16.44001007080078 disk.sda1.out‚0 disk.sda3.in‚ 0.0546875 disk.dm-1.out‚ 7577.046875 disk.drbd2266.out‚0% disk./.usageGB‚225.50426483154297 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚ 364.609375 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.0546875% proc.memory.availableMB‚ 1023.5625* disk./.usagePercent‚15.133780423525758 disk.rbd1.out‚13.34228515625 disk.sda2.out‚03>¹ ‘¹‚(:† „ ãÔíÌ“® gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚121.625' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.325 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0_[xº‚(:í „ ãÔ»ÞÇ° disk.dm-0.out‚23817.89453125 disk.rbd1.out‚13.34228515625 disk.nvme0n1.in‚ 365.69140625 disk.sda.in‚ 0.0546875 disk.dm-0.in‚ 365.69140625 disk.sda.out‚ 7578.00390625 disk.dm-1.out‚ 7578.00390625 cpu‚25.35720436217591 network.sent‚ 1.2522818e+07 network.recv‚ 1.556668e+06 disk.sda3.in‚ 0.0546875% disk.nvme0c0n1.out‚23817.89453125 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2266.out‚0) disk./.usagePercent‚15.13386516201959$ disk./.usageGB‚225.5055274963379 disk.drbd2080.out‚0# disk.nvme0n1.out‚23817.89453125) proc.memory.availableMB‚ 1024.05078125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda3.out‚ 7578.00390625" disk.nvme0c0n1.in‚ 365.69140625) proc.memory.percent‚16.44001007080078 disk.sda1.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.0546875$ memory_percent‚93.74969005584717 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.sda1.out‚0 proc.memory.rssMB‚ 2693.53125 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0AéòÀ2»- _wandbruntime‚4251  _runtime‚4251¹®´}ɼà % _timestamp‚1.7883996263632987e+09 _runtime‚4251.000138706 _step‚432 ' lr-AdamW/pg1‚0.00014835307533768362 ' lr-AdamW/pg2‚0.00014835307533768362 trainer/global_step‚5299•ÕÎ½È trainer/global_step‚5299 % _timestamp‚1.7883996263632987e+09 _runtime‚4251.000138706 _step‚432 ' lr-AdamW/pg1‚0.00014835307533768362 ' lr-AdamW/pg2‚0.00014835307533768362°ÝpY2¾- _wandbruntime‚4251  _runtime‚4251'Cc·''¿ØO L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7602292518400 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 107091221 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚156766967760896 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 107091221 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7472475633664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚165792229668352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7602084030976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4251.000996786 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69800889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚158189937149952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7472644633600 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70850930 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69800889 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69800891 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚149471043467492 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156766780758016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107111259 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7472475633664 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚149294305124352 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7451108599012 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 176892110 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚158189913209856 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7472475633664 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚165792229668352 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7602084030976 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚4251.000996786 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚69800889 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚158189937149952 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7472644633600 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚70850930 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚69800889 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚69800891 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚149471043467492 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚156766780758016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 107111259 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7472475633664 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚149294305124352 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7451108599012 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 176892110 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚158189913209856±ÉÅ Ù2Á- _wandbruntime‚4251  _runtime‚4251ËkkËé&ÂãM ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69813548 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40751683 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107111008 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚165822564796416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚156795443260928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 176924556 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚4251.686621424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚149321592492032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7883996270491252e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚149321574489088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚158218850098176 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70863092 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 176924556 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7473868771840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚165822323993600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚158218826158080 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107131072 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40751681 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107131072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚149498375814580 _step‚434 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69813550 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156795630277120 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7474037785088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176944622 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7603497835520 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69813548 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7473868771840 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚149321574489088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70863623 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30111940 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7452332642152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚149498193653608 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30111411 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚142045861011456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7452497763764 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199—±7¬î&ÃèM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 176944622 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚149321592492032 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7883996270491252e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚149321574489088 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚158218850098176 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚70863092 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 176924556 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7473868771840 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚165822323993600 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚158218826158080 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 107131072 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚40751681 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 107131072 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚149498375814580 _step‚434 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚69813550 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚156795630277120 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7474037785088 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 176944622 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7603497835520 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚69813548 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7473868771840 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚149321574489088 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚70863623 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30111940 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7452332642152 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚149498193653608 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30111411 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚142045861011456 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7452497763764 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚69813548 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚40751683 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 107111008 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚165822564796416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚156795443260928 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 176924556 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚4251.686621424 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚158199920317440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚150837674402700 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 trainer/global_step‚5349 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70433839 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚158200107320320 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 % _timestamp‚1.7883996625029848e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108103961 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30379918 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7542315956224 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178517762 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚158200107320320 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108083923 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚158199920317440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159635584561152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70433841 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167308519720960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41113154 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108083923 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚150657773361152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚150657791364096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7542146956288 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178537802 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30379375 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7520415296016 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7673171781632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71493074 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71492529 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚167308756342784 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159635560621056 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚143317076945920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70433839 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0Ã÷å!ä'ÐÞO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚143317076945920 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚70433839 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _runtime‚4287.140051933 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚158199920317440 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚150837674402700 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 trainer/global_step‚5349 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚70433839 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚158200107320320 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 % _timestamp‚1.7883996625029848e+09 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 108103961 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30379918 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7542315956224 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 178517762 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚158200107320320 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 108083923 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚158199920317440 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚159635584561152 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚70433841 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚167308519720960 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41113154 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 108083923 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚150657773361152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚150657791364096 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7542146956288 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 178537802 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚30379375 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7520415296016 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7673171781632 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚71493074 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚71492529 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚167308756342784 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚159635560621056µJщ12Ñ- _wandbruntime‚4287  _runtime‚4287×jààÏÒâM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30385455 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚158228769836544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70446500 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178570274 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚150685060728832 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚159664473569280 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7674591736320 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚150865006749788 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚143343020128256 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178550208 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41120533 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71505465 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143343037167616 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70446498 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108103710 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178550208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178570274 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70446500 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚438 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7674379067904 _runtime‚4287.823271118 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚150685042725888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7543540094464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159664497509376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7543540094464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚167339089245696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ¯^’äOADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108103710 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7521804460560 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚158228582820352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41120531 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70446498qÔãí&ÓçM EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 108103710 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7521804460560 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚158228582820352 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41120531 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚70446498 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30385455 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚158228769836544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚70446500 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 178570274 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚150685060728832 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚159664473569280 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7674591736320 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚150865006749788 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚143343020128256 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 178550208 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1003 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41120533 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚71505465 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚143343037167616 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚70446498 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 108103710 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 178550208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 178570274 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚70446500 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚438 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7674379067904 _runtime‚4287.823271118 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚150685042725888 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7543540094464 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚159664497509376 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7543540094464 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚167339089245696 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0¶ÖH"î2Ô- _wandbruntime‚4287  _runtime‚4287LŠ°RZÕÔ 0loss/FBt1/assignment_loss‚0.4759168790444619 /loss/FRt1/detection_loss‚2.1940336059440266 _step‚439 % _timestamp‚1.7883996631875622e+09 0loss/FBt2/assignment_loss‚0.2148178586179256 /loss/FRt1/assignment_loss‚3.161880812861703 1loss/SRqqt3/detection_loss‚1.8512639580470212 _runtime‚4287.824309187 0loss/FRt2/assignment_loss‚4.4618457302902685 2loss/SRqqt4/assignment_loss‚0.9505486160852625 1loss/SRqqt4/detection_loss‚1.1909910616938209 epoch‚4 2loss/SRqqt1/assignment_loss‚1.3542063957809105 .loss/FRt3/detection_loss‚2.359095562249422 .loss/FBt1/detection_loss‚2.139376676744885 /loss/FBt3/detection_loss‚2.4286658639709153 %loss/total_loss‚46.16085298650624 0loss/FRt3/assignment_loss‚3.1737212583422663 0loss/FRt4/assignment_loss‚2.0729687702655792 0loss/FBt3/assignment_loss‚0.4761682976301434 /loss/FRt2/detection_loss‚2.8130786075736536 /loss/FRt4/detection_loss‚1.9545542341470719 1loss/SRqqt1/detection_loss‚1.8234927145683246 .loss/FBt2/detection_loss‚4.124140691757202 /loss/FBt4/detection_loss‚0.8941750536457851 2loss/SRqqt2/assignment_loss‚1.3452398283407092 2loss/SRqqt3/assignment_loss‚1.5720652954055454 0loss/FBt4/assignment_loss‚0.1779283997776789 0loss/SRqqt2/detection_loss‚2.950676813721657 trainer/global_step‚5349†CÒÒ_ÖÙ 0loss/FRt4/assignment_loss‚2.0729687702655792 0loss/FBt3/assignment_loss‚0.4761682976301434 /loss/FRt2/detection_loss‚2.8130786075736536 /loss/FRt4/detection_loss‚1.9545542341470719 1loss/SRqqt1/detection_loss‚1.8234927145683246 .loss/FBt2/detection_loss‚4.124140691757202 /loss/FBt4/detection_loss‚0.8941750536457851 2loss/SRqqt2/assignment_loss‚1.3452398283407092 2loss/SRqqt3/assignment_loss‚1.5720652954055454 0loss/FBt4/assignment_loss‚0.1779283997776789 0loss/SRqqt2/detection_loss‚2.950676813721657 trainer/global_step‚5349 0loss/FBt1/assignment_loss‚0.4759168790444619 /loss/FRt1/detection_loss‚2.1940336059440266 _step‚439 % _timestamp‚1.7883996631875622e+09 0loss/FBt2/assignment_loss‚0.2148178586179256 /loss/FRt1/assignment_loss‚3.161880812861703 1loss/SRqqt3/detection_loss‚1.8512639580470212 _runtime‚4287.824309187 0loss/FRt2/assignment_loss‚4.4618457302902685 2loss/SRqqt4/assignment_loss‚0.9505486160852625 1loss/SRqqt4/detection_loss‚1.1909910616938209 epoch‚4 2loss/SRqqt1/assignment_loss‚1.3542063957809105 .loss/FRt3/detection_loss‚2.359095562249422 .loss/FBt1/detection_loss‚2.139376676744885 /loss/FBt3/detection_loss‚2.4286658639709153 %loss/total_loss‚46.16085298650624 0loss/FRt3/assignment_loss‚3.1737212583422663· 2øђׂ(:‡ ± ãÔ㡤® gpu.0.gpu‚42.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚118.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7332 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‹3"Ø‚(:ö ± ãÔ†×í°% disk./.usageGB‚225.50821685791016 disk.sda2.out‚0 disk.rbd0.out‚0# disk.dm-0.out‚23824.62255859375) proc.memory.percent‚16.44001007080078 disk.sda3.in‚ 0.0546875 disk.sda3.out‚ 7579.9453125 disk.drbd2266.out‚0 disk.dm-0.in‚ 366.41015625 disk.sda1.in‚0 cpu‚25.353996926047262 disk.drbd1716.in‚0 disk.dm-1.out‚ 7579.9453125 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0$ memory_percent‚93.75560283660889) proc.memory.availableMB‚ 1023.08203125 network.sent‚ 1.2642884e+07 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 366.41015625 proc.cpu.threads‚17 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.rbd1.out‚13.34228515625 network.recv‚ 1.570975e+06* disk./.usagePercent‚15.134045647331229 disk.sda.in‚ 0.0546875 disk.dm-1.in‚ 0.0546875 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 366.41015625( disk.nvme0c0n1.out‚23824.62255859375& disk.nvme0n1.out‚23824.62255859375 disk.drbd1716.out‚0 disk.drbd2266.in‚0 proc.memory.rssMB‚ 2693.53125 disk.sda.out‚ 7579.9453125sLnÙ‚(:… À ãÔÞªœ® gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚123.51' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.702 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Æ7t'‚Ú‚(:÷ À ãÔňư disk.sda.in‚ 0.0546875 disk.sda2.out‚0 disk.drbd2080.in‚0" disk.nvme0c0n1.in‚ 366.41015625& disk.nvme0n1.out‚23824.91552734375 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0$ memory_percent‚93.75524520874023 network.recv‚ 1.577145e+06 disk.nvme0n1.in‚ 366.41015625 disk.dm-1.in‚ 0.0546875 disk.drbd2265.in‚0 disk.rbd1.in‚0' proc.memory.availableMB‚ 1023.140625 proc.memory.rssMB‚ 2693.53125) proc.memory.percent‚16.44001007080078 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.sda3.out‚ 7580.60546875 network.sent‚ 1.2751328e+07 disk.sda.out‚ 7580.60546875 disk.dm-1.out‚ 7580.60546875 disk.rbd0.in‚0 disk.rbd1.out‚13.34228515625 disk.rbd0.out‚0% disk./.usageGB‚225.50823974609375 disk.dm-0.in‚ 366.41015625 disk.sda1.in‚0 disk.sda3.in‚ 0.0546875 cpu‚25.352889879451425 proc.cpu.threads‚17* disk./.usagePercent‚15.134047183376437( disk.nvme0c0n1.out‚23824.91552734375 disk.drbd2080.out‚0# disk.dm-0.out‚23824.91552734375 disk.sda1.out‚0Q2p™Û‚(:’ Ï ãÔÁ¾Ê° gpu.0.gpu‚32.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚124.287' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.857400000000002 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X2ÓèƒÜ‚(:ø Ï ãÔçÆײ proc.memory.rssMB‚ 2693.53125) proc.memory.percent‚16.44001007080078% disk./.usageGB‚225.50818252563477& disk.nvme0n1.out‚23825.41162109375 disk.drbd1716.in‚0 network.sent‚ 1.2757615e+07 disk.rbd1.in‚0 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 7581.07421875( proc.memory.availableMB‚ 1023.0546875 disk.rbd0.out‚0 disk.rbd1.out‚13.34228515625( disk.nvme0c0n1.out‚23825.41162109375 disk.dm-1.in‚ 0.0546875 disk.drbd2266.in‚0$ memory_percent‚93.75576972961426 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda3.in‚ 0.0546875" disk.nvme0c0n1.in‚ 366.41015625 disk.nvme0n1.in‚ 366.41015625 disk.sda.in‚ 0.0546875* disk./.usagePercent‚15.134043343263423 disk.dm-0.in‚ 366.41015625# disk.dm-0.out‚23825.41162109375 cpu‚25.351875858207386 disk.sda1.in‚0 disk.sda3.out‚ 7581.07421875 disk.sda2.in‚0 disk.drbd2265.in‚0 network.recv‚ 1.581886e+06 disk.drbd2080.in‚0 disk.sda.out‚ 7581.07421875íË.¤2Ý- _wandbruntime‚4323  _runtime‚4323È6à„ÇÞÁ trainer/global_step‚5399 % _timestamp‚1.7883996987153738e+09 _runtime‚4323.352309642 _step‚440 & lr-AdamW/pg1‚0.0001454658243419546 & lr-AdamW/pg2‚0.0001454658243419546Ù²UðÌßÆ & lr-AdamW/pg1‚0.0001454658243419546 & lr-AdamW/pg2‚0.0001454658243419546 trainer/global_step‚5399 % _timestamp‚1.7883996987153738e+09 _runtime‚4323.352309642 _step‚440¸©]Ó¨2à- _wandbruntime‚4323  _runtime‚4323¡²½÷Þ'áØO H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚168825046192128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159633059876864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71066791 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109076625 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚152021259600896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7611818278912 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180143414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71066791 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚159633246879744 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚441 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109096663 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚152021259600896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚152021241597952 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1003 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚168825287008256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7590052235828 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30651294 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚5399 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7611818278912 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109096663 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _runtime‚ 4323.35313556 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7611987278848 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161081231972352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚571 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚152204123177144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0Áé×&~ âÝO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71066791 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 109076625 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚152021259600896 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7611818278912 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 180143414 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚71066791 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚159633246879744 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚441 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 109096663 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚152021259600896 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚152021241597952 H?DeviceStatsMonitoÐTùÎer.on_train_batch_start/inactive_split.all.peak‚1003 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚168825287008256 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7590052235828 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30651294 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚5399 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7611818278912 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 109096663 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _runtime‚ 4323.35313556 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7611987278848 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚161081231972352 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚571 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚152204123177144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚168825046192128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚159633059876864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚982 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416¹íj2ã- _wandbruntime‚4324  _runtime‚4324»ùCé&äãM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180195926 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7613380430336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚161110120980480 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109116476 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚144640196284416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚161110144920576 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚168855371670016 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7613380430336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71079448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180195926 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109096412 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41489383 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30656663 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚152048528965632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71079450 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚507 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚144640179245056 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7591441400580 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30656156 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71079448 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 trainer/global_step‚5399 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚152231455524024 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41489381 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚159661909395968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚152048510962688 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72145537 % _timestamp‚1.7883996994000177e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180175860 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7745250689536 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7591276278968 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7745469649408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71079450 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚152048510962688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159661722379776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚509m+@î&åèM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7745469649408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71079450 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚152048510962688 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚159661722379776 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚509 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 180195926 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7613380430336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚161110120980480 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 109116476 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚144640196284416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚161110144920576 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚168855371670016 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7613380430336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚71079448 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 180195926 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 109096412 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41489383 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚30656663 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚152048528965632 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71079450 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚507 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚144640179245056 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7591441400580 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30656156 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71079448 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 trainer/global_step‚5399 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚152231455524024 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41489381 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚159661909395968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚152048510962688 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72145537 % _timestamp‚1.7883996994000177e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 180175860 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7745250689536 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7591276278968 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚161066199436288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41850854 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚153570936273116 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110089365 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7659524054236 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚30923017 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚170341800356352 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚153384709834752 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7681489601536 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚170341567928832 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚153570754112352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚161066199436288 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110069327 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚161066386439168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41850856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚153384727837696 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚161066386439168 _runtime‚4359.327920175 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7814712485376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _step‚445 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚561 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7814920972800 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110089365 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72773310 trainer/global_step‚5449 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71699741 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7659358932832 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚161066199436288 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 110069327 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚161066386439168 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚41850856 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚153384727837696 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚161066386439168 _runtime‚4359.327920175 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7814712485376 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _step‚445 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚561 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7814920972800 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 110089365 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚72773310 trainer/global_step‚5449 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚71699741 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7659358932832 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚161066199436288 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚41850854 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚153570936273116 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 110089365 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7659524054236½o+À:2ó- _wandbruntime‚4360  _runtime‚4360ÝŽŸÌé&ôãM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 181821578 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚568 _runtime‚4360.006788477 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚161095048955392 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181801512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41858231 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110109178 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚162555792331776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7816130618368 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72786086 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71712400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚170372135618560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚145937338361856 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71712400 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚161094861939200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚145937355401216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7660748097376 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71712398 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚161095048955392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181801512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚162555768391680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7883997353695283e+09 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7660913218988 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110089114 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚153411979199488 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚153411979199488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚446 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7683051752960 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚170371899010048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181821578 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72786654 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚566 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚153598086459232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41858233 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30927855 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚161095048955392 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 181801512 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚41858231 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚982 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 110109178 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_p"ëÎÐÎ"ool.peak‚ 410712064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚162555792331776 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7816130618368 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚72786086 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚71712400 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚170372135618560 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚145937338361856 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚71712400 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚161094861939200 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚145937355401216 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7660748097376 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚71712398 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚161095048955392 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 181801512 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚162555768391680 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7883997353695283e+09 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7660913218988 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 110089114 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚153411979199488 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚153411979199488 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚446 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7683051752960 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚170371899010048 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 181821578 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚72786654 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚566 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚153598086459232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚41858233 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚30927855 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217¾ó²kå2ö- _wandbruntime‚4360  _runtime‚4360 Ê#?Y÷Ó 0loss/FRt1/assignment_loss‚3.3484406349835574 1loss/SRqqt3/assignment_loss‚1.573552620668824 .loss/FBt3/detection_loss‚3.438498842716217 _runtime‚ 4360.00785514 2loss/SRqqt4/assignment_loss‚1.0753101365194775 0loss/FBt3/assignment_loss‚0.2048566360026541 .loss/FRt3/detection_loss‚2.374118219201381 /loss/FBt2/detection_loss‚2.7332187712192537 epoch‚4 _step‚447 /loss/FRt4/assignment_loss‚1.329682661037819 0loss/FBt2/assignment_loss‚0.7455543830874376 /loss/FRt1/detection_loss‚2.1200519369708166 1loss/SRqqt1/detection_loss‚1.9847606733441352 1loss/SRqqt3/detection_loss‚1.6579388551987135 .loss/FBt1/detection_loss‚2.730924717404626 2loss/SRqqt1/assignment_loss‚1.6479878349436654 /loss/FBt4/detection_loss‚0.7671966911328412 2loss/SRqqt2/assignment_loss‚2.0776828729680608 1loss/FBt4/assignment_loss‚0.26012613901633824 1loss/SRqqt2/detection_loss‚2.5395915806293488 1loss/SRqqt4/detection_loss‚1.0810464581763242 &loss/total_loss‚47.181005435903046 trainer/global_step‚5449 0loss/FBt1/assignment_loss‚0.3012935956712681 /loss/FRt2/assignment_loss‚4.380812177291284 0loss/FRt3/assignment_loss‚3.6628646972851877 /loss/FRt2/detection_loss‚3.2063706517219543 /loss/FRt4/detection_loss‚1.9391236487118637 $ _timestamp‚1.788399735371116e+098þ‡#^øØ 2loss/SRqqt4/assignment_loss‚1.0753101365194775 0loss/FBt3/assignment_loss‚0.2048566360026541 .loss/FRt3/detection_loss‚2.374118219201381 /loss/FBt2/detection_loss‚2.7332187712192537 epoch‚4 _step‚447 /loss/FRt4/assignment_loss‚1.329682661037819 0loss/FBt2/assignment_loss‚0.7455543830874376 /loss/FRt1/detection_loss‚2.1200519369708166 1loss/SRqqt1/detection_loss‚1.9847606733441352 1loss/SRqqt3/detection_loss‚1.6579388551987135 .loss/FBt1/detection_loss‚2.730924717404626 2loss/SRqqt1/assignment_loss‚1.6479878349436654 /loss/FBt4/detection_loss‚0.7671966911328412 2loss/SRqqt2/assignment_loss‚2.0776828729680608 1loss/FBt4/assignment_loss‚0.26012613901633824 1loss/SRqqt2/detection_loss‚2.5395915806293488 1loss/SRqqt4/detection_loss‚1.0810464581763242 &loss/total_loss‚47.181005435903046 trainer/global_step‚5449 0loss/FBt1/assignment_loss‚0.3012935956712681 /loss/FRt2/assignment_loss‚4.380812177291284 0loss/FRt3/assignment_loss‚3.6628646972851877 /loss/FRt2/detection_loss‚3.2063706517219543 /loss/FRt4/detection_loss‚1.9391236487118637 $ _timestamp‚1.788399735371116e+09 0loss/FRt1/assignment_loss‚3.3484406349835574 1loss/SRqqt3/assignment_loss‚1.573552620668824 .loss/FBt3/detection_loss‚3.438498842716217 _runtime‚ 4360.00785514¿‘¬¯Jù‚(:’ ü ãÔƒž»® gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚143.998' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.799599999999998 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0nº|ú‚(:ñ ü ãÔ¦žŽ± disk.drbd2080.out‚0 cpu‚25.348743258028442 disk.sda3.out‚ 7582.671875" disk.nvme0c0n1.in‚ 366.41015625 disk.dm-1.out‚ 7582.671875* disk./.usagePercent‚15.133944524355206% disk./.usageGB‚225.50671005249023 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 366.41015625 disk.drbd2265.out‚0) proc.memory.percent‚16.44001007080078 proc.cpu.threads‚17 disk.sda1.out‚0( disk.nvme0c0n1.out‚23826.72998046875 disk.sda.in‚ 0.0546875 disk.dm-1.in‚ 0.0546875' proc.memory.availableMB‚ 1022.734375 proc.memory.rssMB‚ 2693.53125 network.sent‚ 1.2877617e+07 disk.sda3.in‚ 0.0546875 disk.rbd1.in‚0& disk.nvme0n1.out‚23826.72998046875 disk.drbd2266.out‚0 disk.drbd2080.in‚0$ memory_percent‚93.75772476196289 disk.dm-0.in‚ 366.41015625 disk.sda1.in‚0 disk.rbd1.out‚13.42822265625 disk.sda.out‚ 7582.671875 disk.drbd1716.in‚0 network.recv‚ 1.595995e+06 disk.drbd2266.in‚0# disk.dm-0.out‚23826.72998046875 disk.drbd1716.out‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0Œ³¥”’û‚(:‡ ‹¡ãÔ«ê´® gpu.0.gpu‚27.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚134.022' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.8044 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‡Bq¼{ü‚(:ð ‹¡ãÔí¡å° disk.rbd0.out‚0 disk.sda1.in‚0 cpu‚25.347729998855378 proc.cpu.threads‚17 disk.drbd2080.out‚0" disk.nvme0c0n1.in‚ 366.41015625 disk.dm-1.out‚ 7583.1015625 disk.drbd2266.out‚0% proc.memory.availableMB‚ 1022.4375) proc.memory.percent‚16.44001007080078$ disk./.usageGB‚225.5067481994629 disk.rbd0.in‚0 disk.nvme0n1.in‚ 366.41015625& disk.nvme0n1.out‚23827.95947265625 disk.dm-1.in‚ 0.0546875 disk.sda1.out‚0 disk.rbd1.out‚13.49072265625 disk.sda2.out‚0 disk.drbd1716.out‚0 network.sent‚ 1.2986251e+07 disk.drbd2265.out‚0 disk.dm-0.in‚ 366.41015625 disk.sda2.in‚0* disk./.usagePercent‚15.133947084430547 disk.sda3.out‚ 7583.1015625 disk.rbd1.in‚0( disk.nvme0c0n1.out‚23827.95947265625 disk.drbd2266.in‚0 network.recv‚ 1.60234e+06 disk.drbd2080.in‚0# disk.dm-0.out‚23827.95947265625 disk.sda.in‚ 0.0546875 disk.sda.out‚ 7583.1015625$ memory_percent‚93.75953674316406 disk.drbd2265.in‚0 disk.sda3.in‚ 0.0546875 disk.drbd1716.in‚0 proc.memory.rssMB‚ 2693.53125!DB‘ý‚(:† š¡ãÔ•”® gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚145.595' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.119 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0EæA{þ‚(:ð š¡ãÔ³çÙ± disk.rbd0.in‚0# disk.dm-0.out‚23828.24072265625 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 366.41015625 disk.drbd1716.in‚0 cpu‚25.346652278700166 disk.rbd0.out‚0 disk.sda3.out‚ 7583.703125 disk.dm-1.out‚ 7583.703125 disk.drbd2265.in‚0$ memory_percent‚93.75505447387695' proc.memory.availableMB‚ 1023.171875* disk./.usagePercent‚15.133944524355206% disk./.usageGB‚225.50671005249023 disk.sda.out‚ 7583.703125 disk.drbd2266.in‚0 disk.drbd2265.out‚0 network.recv‚ 1.60503e+06" disk.nvme0c0n1.in‚ 366.41015625 disk.sda2.out‚0 disk.dm-1.in‚ 0.0546875 disk.drbd2266.out‚0 network.sent‚ 1.2990738e+07 disk.drbd2080.in‚0 disk.dm-0.in‚ 366.41015625 disk.sda3.in‚ 0.0546875 disk.rbd1.in‚0 disk.rbd1.out‚13.49072265625 disk.sda2.in‚0 disk.drbd1716.out‚0( disk.nvme0c0n1.out‚23828.24072265625 proc.memory.rssMB‚ 2693.53125) proc.memory.percent‚16.44001007080078 disk.drbd2080.out‚0& disk.nvme0n1.out‚23828.24072265625 disk.sda.in‚ 0.0546875 proc.cpu.threads‚17G¹[2ÿ- _wandbruntime‚4395  _runtime‚4395$àx1Ç€Á ' lr-AdamW/pg1‚0.00014254692133242153 ' lr-AdamW/pg2‚0.00014254692133242153 trainer/global_step‚5499 $ _timestamp‚1.788399770772855e+09 _runtime‚ 4395.40967708 _step‚448¿YæqÌÆ ' lr-AdamW/pg2‚0.00014254692133242153 trainer/global_step‚5499 $ _timestamp‚1.788399770772855e+09 _runtime‚ 4395.40967708 _step‚448 ' lr-AdamW/pg1‚0.00014254692133242153À F™e2‚- _wandbruntime‚4395  _runtime‚4395ÎûèÝ'ƒ×O aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚163972502854656 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72332691 trainer/global_step‚5499 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183394718 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7751329924096 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72332689 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚147208554296320 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚171858306010624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚154748178071552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31195933 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42219706 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚147208571335680 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7751160924160 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111062029 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚163972526794752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚449 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚162499525998592 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚162499338995712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚154748196074496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202195968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7751329924096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226136064 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚154937567208324 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111082067 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42219704ƒøŸìÓ„ÜO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 183414758 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _runtime‚4395.410705079 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7751329924096 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72332689 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚147208554296320 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚171858306010624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚154748178071552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31195933 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42219706 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚147208571335680 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 7751160924160 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 111062029 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚163972526794752 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚449 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚162499525998592 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚162499338995712 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚154748196074496 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202195968 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7751329924096 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226136064 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚154937567208324 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 111082067 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42219704 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚163972502854656 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚72332691 trainer/global_step‚5499 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 183394718 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183447230 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73428694 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7752723075584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚164001415802880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7886990841856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚154775447436288 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7752723075584 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111101880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚147234497478656 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7730219915784 _step‚450 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚162528188514816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31201611 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31201088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7752554062336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72345348 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73428169 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 $ _timestamp‚1.788399771453771e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 trainer/global_step‚5499 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72345348 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚154775465439232 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚164001439742976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248–:Ü«í&‡çM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 183427164 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 111081816 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚162528188514816 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚154775465439232 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 183447230 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚73428694 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7752723075584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚164001415802880 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 7886990841856 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚154775447436288 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7752723075584 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 111101880 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚147234497478656 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7730219915784 _step‚450 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚162528188514816 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31201611 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31201088 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7752554062336 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚72345348 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚73428169 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 $ _timestamp‚1.788399771453771e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 trainer/global_step‚5499 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72345348 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚154775465439232 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚164001439742976 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248Â)o72ˆ- _wandbruntime‚4396  _runtime‚4396Ï’§\‰Ö 0loss/SRqqt1/detection_loss‚1.775543530259216 _step‚451 /loss/FRt4/assignment_loss‚1.551349836587906 /loss/FRt2/detection_loss‚3.2545801446988034 1loss/SRqqt3/detection_loss‚1.7987305748586853 epoch‚4 % _timestamp‚1.7883997714554634e+09 2loss/SRqqt1/assignment_loss‚1.6231662552607686 0loss/FBt3/assignment_loss‚0.3438017606678841 0loss/SRqqt2/detection_loss‚3.026677079822706 2loss/SRqqt3/assignment_loss‚1.7190097933635116 0loss/FBt1/assignment_loss‚0.9501105079427361 /loss/FBt4/detection_loss‚0.6654900307426366 0loss/FRt1/assignment_loss‚3.5362943976612415 /loss/FRt2/assignment_loss‚4.106737998815683 1loss/FBt4/assignment_loss‚0.13637284044912898 /loss/FRt4/detection_loss‚1.6310964050889014 /loss/FBt2/detection_loss‚2.8337543681263924 &loss/total_loss‚46.581145224880544 2loss/SRqqt2/assignment_loss‚2.1487401432317235 0loss/FBt2/assignment_loss‚0.5201711570844054 /loss/FRt1/detection_loss‚1.9642293478472759 /loss/FRt3/detection_loss‚2.5391391944140196 0loss/FRt3/assignment_loss‚2.7455173563212156 1loss/SRqqt4/detection_loss‚1.1305842104678352 /loss/FBt1/detection_loss‚1.7566774398088456 trainer/global_step‚5499 _runtime‚4396.092205426 2loss/SRqqt4/assignment_loss‚0.9095875514360766 .loss/FBt3/detection_loss‚3.913783299922943wmãaŠÛ 2loss/SRqqt1/assignment_loss‚1.6231662552607686 0loss/FBt3/assignment_loss‚0.3438017606678841 0loss/SRqqt2/detection_loss‚3.026677079822706 2loss/SRqqt3/assignment_loss‚1.7190097933635116 0loss/FBt1/assignment_loss‚0.9501105079427361 /loss/FBt4/detection_loss‚0.6654900307426366 0loss/FRt1/assignment_loss‚3.5362943976612415 /loss/FRt2/assignment_loss‚4.106737998815683 1loss/FBt4/assignment_loss‚0.13637284044912898 /loss/FRt4/detection_loss‚1.6310964050889014 /loss/FBt2/detection_loss‚2.8337543681263924 &loss/total_loss‚46.581145224880544 2loss/SRqqt2/assignment_loss‚2.1487401432317235 0loss/FBt2/assignment_loss‚0.5201711570844054 /loss/FRt1/detection_loss‚1.9642293478472759 /loss/FRt3/detection_loss‚2.5391391944140196 0loss/FRt3/assignment_loss‚2.7455173563212156 1loss/SRqqt4/detection_loss‚1.1305842104678352 /loss/FBt1/detection_loss‚1.7566774398088456 trainer/global_step‚5499 _runtime‚4396.092205426 2loss/SRqqt4/assignment_loss‚0.9095875514360766 .loss/FBt3/detection_loss‚3.913783299922943 0loss/SRqqt1/detection_loss‚1.775543530259216 _step‚451 /loss/FRt4/assignment_loss‚1.551349836587906 /loss/FRt2/detection_loss‚3.2545801446988034 1loss/SRqqt3/detection_loss‚1.7987305748586853 epoch‚4 % _timestamp‚1.7883997714554634e+09Ã<ŽÆ¬‘‹‚(:† ©¡ãԱș® gpu.0.gpu‚23.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚116.485' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.297 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Í‚ö{Œ‚(:ð ©¡ãÔ›×Ò° disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.dm-0.out‚23829.0595703125 disk.rbd1.out‚13.52294921875 disk.dm-0.in‚ 366.41015625' disk.nvme0c0n1.out‚23829.0595703125 disk.nvme0n1.in‚ 366.41015625 disk.dm-1.out‚ 7584.46484375 disk.drbd2266.out‚0$ memory_percent‚93.75686645507812 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 7584.46484375 disk.sda2.in‚0 disk.drbd1716.in‚0 proc.memory.rssMB‚ 2693.53125 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.sda.in‚ 0.0546875) proc.memory.percent‚16.44001007080078 disk.rbd1.in‚0$ proc.memory.availableMB‚1022.875 cpu‚25.34571760746691 network.recv‚ 1.609213e+06 disk.drbd2265.out‚0 disk.sda1.out‚0% disk.nvme0n1.out‚23829.0595703125 disk.sda2.out‚0 disk.drbd1716.out‚0 proc.cpu.threads‚17 disk.sda.out‚ 7584.46484375 network.sent‚ 1.2996742e+07* disk./.usagePercent‚15.133944524355206% disk./.usageGB‚225.50671005249023 disk.sda3.in‚ 0.0546875" disk.nvme0c0n1.in‚ 366.41015625 disk.dm-1.in‚ 0.0546875ñ‚(:’ ¸¡ãÔÍ„À® gpu.0.gpu‚34.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚126.468' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.293599999999998 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%™Rþ}Ž‚(:ò ¸¡ãÔ¼ï° disk.sda.out‚ 7584.8671875 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.sda2.in‚0 disk.dm-0.in‚ 366.41015625 disk.sda1.in‚0 disk.nvme0n1.in‚ 366.41015625 proc.memory.rssMB‚ 2693.53125 proc.cpu.threads‚17 disk.drbd2080.in‚0' disk.nvme0c0n1.out‚23829.3095703125 disk.dm-1.out‚ 7584.8671875( proc.memory.availableMB‚ 1022.8671875) proc.memory.percent‚16.44001007080078% disk./.usageGB‚225.50671005249023 disk.sda3.in‚ 0.0546875 disk.rbd1.in‚0 network.sent‚ 1.3105033e+07 disk.rbd0.out‚0" disk.dm-0.out‚23829.3095703125 disk.rbd1.out‚13.52294921875" disk.nvme0c0n1.in‚ 366.41015625 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.0546875 cpu‚25.344718130079805 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 7584.8671875 disk.sda.in‚ 0.0546875 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.sda1.out‚0% disk.nvme0n1.out‚23829.3095703125 disk.sda2.out‚0$ memory_percent‚93.75691413879395 network.recv‚ 1.615731e+06* disk./.usagePercent‚15.133944524355206 disk.drbd2266.out‚0¦U 2- _wandbruntime‚4431  _runtime‚4431‰`_–Éà ' lr-AdamW/pg1‚0.00014107635967633344 ' lr-AdamW/pg2‚0.00014107635967633344 trainer/global_step‚5549 % _timestamp‚1.7883998067169275e+09 _runtime‚4431.353761531 _step‚452›ß<þÎ‘È _step‚452 ' lr-AdamW/pg1‚0.00014107635967633344 ' lr-AdamW/pg2‚0.00014107635967633344 trainer/global_step‚5549 % _timestamp‚1.7883998067169275e+09 _runtime‚4431.353761531ÄY}éï2’- _wandbruntime‚4431  _runtime‚4431©Óãj“ØO [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7821001246720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚163932665558016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156111664311296 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165418174205952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72965639 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7956634954752 $ _timestamp‚1.788399806717595e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7820832246784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156111646308352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚173374809160704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7956426467328 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚156304198143532 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112074769 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚163932665558016 QCDeviÁÔT¯t!ceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚173374576733184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚163932478555136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚156304015982768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74058956 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚148505730452480 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚163932478555136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4431.354615452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72965641 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚156111646308352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112054731 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚5549 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7821001246720 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112054731 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165418150265856 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156111664311296 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72965639 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74058415 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42588556 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112074769 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31469861 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 185020370 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31470400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚148505713413120 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185040410 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42588554 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7798467691052 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160ìã'”ÝO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚4431.354615452 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚72965641 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚156111646308352 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 112054731 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚5549 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7821001246720 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 112054731 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚165418150265856 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚156111664311296 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚72965639 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74058415 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42588556 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 112074769 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31469861 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 185020370 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31470400 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚148505713413120 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 185040410 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42588554 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7798467691052 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7821001246720 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚163932665558016 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚156111664311296 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚165418174205952 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚72965639 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 7956634954752 $ _timestamp‚1.788399806717595e+09 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7820832246784 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚156111646308352 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚173374809160704 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 7956426467328 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚156304198143532 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 112074769 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚163932665558016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚173374576733184 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚163932478555136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚156304015982768 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚74058956 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚148505730452480 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚163932478555136 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0ÅÒ #Ã2•- _wandbruntime‚4432  _runtime‚4432‚¡®é&–ãM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚148531673634816 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185072882 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 185052816 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173404902045184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7822394398208 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7822225384960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42595931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72978298 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚173405142848000 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156138933676032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚165447087154176 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112074518 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7822394398208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74071291 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31475877 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112074518 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7958055693824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156138933676032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72978300 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7883998073909588e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7822225384960 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 7958055693824 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚156138933676032 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚72978300 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7883998073909588e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7822225384960 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 185072882 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 185052816 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚173404902045184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7822394398208 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7822225384960 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42595931 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚72978298 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚173405142848000 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚156138933676032 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚165447087154176 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 112074518 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7822394398208 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74071291 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31475877 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 112074518 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113067471 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7890672569344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚157475114545152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚165365618114560 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73598591 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚165365805117440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73598589 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚174891188842496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113047433 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚166863797677056 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 trainer/global_step‚5599 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113067471 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42957404 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31743714 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7867939509460 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚157475114545152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74700590 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42957406 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31743186 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7867774388056 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚165365805117440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8027608041472 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149802872529920 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 % _timestamp‚1.7883998431131086e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 _runtime‚4467.750178244 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚165365618114560 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚157670829078740 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217©Ì³@'¤ÞO =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚165365618114560 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚157670829078740 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 7890672569344 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚73598589 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚166863821617152 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8027391165440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚157670646917976 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚149802889569280 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 113067471 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 7890672569344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚157475114545152 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚165365618114560 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚73598591 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚165365805117440 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚73598589 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚528 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚174891188842496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 113047433 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚166863797677056 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 trainer/global_step‚5599 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 113067471 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚42957404 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚31743714 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 7867939509460 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚157475114545152 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚74700590 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚42957406 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚31743186 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 7867774388056 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚165365805117440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8027608041472 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚149802872529920 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 % _timestamp‚1.7883998431131086e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_st^E¡4¤art/inactive_split_bytes.all.peak‚ 269877760 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 _runtime‚4467.750178244ÉTøö2¥- _wandbruntime‚4468  _runtime‚4468‹×Áé&¦ãM XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7891896707584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7869163552600 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7892065720832 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74713400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157697979264856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149828832751616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113087284 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚165394467633664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42964781 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚4468.441235297 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186698534 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚157502383909888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113067220 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚166892734565376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚166892710625280 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186678468 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7892065720832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186678468 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8028810138624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74713905 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚174921520763904 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 trainer/global_step‚5599 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73611248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73611248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8029020709888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7891896707584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7883998438039386e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113067220 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚165394467633664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚165394280617472 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚157502383909888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚149828815712256 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73611250 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚503 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113087284 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚157698161425828 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7869328674212 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31749122 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73611250 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚157502401912832 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186698534 _step‚458 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31748619 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚505/ˆ6î&§èM H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚505 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 7891896707584 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7869163552600 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7892065720832 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚74713400 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚157697979264856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚149828832751616 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 113087284 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚165394467633664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚42964781 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚4468.441235297 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 186698534 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚157502383909888 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 113067220 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚166892734565376 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚166892710625280 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 186678468 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7892065720832 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 186678468 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8028810138624 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚74713905 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚174921520763904 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 trainer/global_step‚5599 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚73611248 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚73611248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8029020709888 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7891896707584 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 % _timestamp‚1.7883998438039386e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 113067220 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚165394467633664 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚165394280617472 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚157502383909888 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚149828815712256 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚73611250 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚503 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 113087284 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚157698161425828 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7869328674212 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚31749122 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚73611250 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚157502401912832 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 186698534 _step‚458 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚31748619 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064ÊÖ Jˆ2¨- _wandbruntime‚4468  _runtime‚4468'}ê[©Õ 1loss/SRqqt3/detection_loss‚2.2187282048165797 %loss/total_loss‚44.74126441296223 % _timestamp‚1.7883998438055658e+09 _runtime‚4468.442637743 _step‚459 0loss/FRt1/assignment_loss‚3.6079213389983544 1loss/SRqqt1/assignment_loss‚1.860201412619188 /loss/FBt1/assignment_loss‚0.381350392854074 1loss/SRqqt1/detection_loss‚1.9064773922336513 epoch‚4 /loss/FRt3/assignment_loss‚2.614431646562392 /loss/FRt2/detection_loss‚2.8434823671976726 1loss/SRqqt2/detection_loss‚2.5576378683890066 /loss/FRt2/assignment_loss‚4.603294618924459 /loss/FBt1/detection_loss‚2.5914423912763596 2loss/SRqqt4/assignment_loss‚1.1341309368610382 /loss/FRt4/detection_loss‚1.6892562965894569 0loss/FRt4/assignment_loss‚1.0804319114520633 0loss/FBt2/assignment_loss‚0.2503095234111627 1loss/FBt3/assignment_loss‚0.20912041311885332 /loss/FRt3/detection_loss‚2.5723318515285367 0loss/SRqqt4/detection_loss‚1.051453963880028 /loss/FBt2/detection_loss‚2.4855756844793047 trainer/global_step‚5599 1loss/SRqqt2/assignment_loss‚2.296090908588902 2loss/SRqqt3/assignment_loss‚1.3097319662570954 1loss/FBt4/assignment_loss‚0.13969848425486991 /loss/FRt1/detection_loss‚2.1405354749697905 /loss/FBt3/detection_loss‚2.5254527818072927 /loss/FBt4/detection_loss‚0.6721765818921003çÏ}`ªÚ 2loss/SRqqt4/assignment_loss‚1.1341309368610382 /loss/FRt4/detection_loss‚1.6892562965894569 0loss/FRt4/assignment_loss‚1.0804319114520633 0loss/FBt2/assignment_loss‚0.2503095234111627 1loss/FBt3/assignment_loss‚0.20912041311885332 /loss/FRt3/detection_loss‚2.5723318515285367 0loss/SRqqt4/detection_loss‚1.051453963880028 /loss/FBt2/detection_loss‚2.4855756844793047 trainer/global_step‚5599 1loss/SRqqt2/assignment_loss‚2.296090908588902 2loss/SRqqt3/assignment_loss‚1.3097319662570954 1loss/FBt4/assignment_loss‚0.13969848425486991 /loss/FRt1/detection_loss‚2.1405354749697905 /loss/FBt3/detection_loss‚2.5254527818072927 /loss/FBt4/detection_loss‚0.6721765818921003 1loss/SRqqt3/detection_loss‚2.2187282048165797 %loss/total_loss‚44.74126441296223 % _timestamp‚1.7883998438055658e+09 _runtime‚4468.442637743 _step‚459 0loss/FRt1/assignment_loss‚3.6079213389983544 1loss/SRqqt1/assignment_loss‚1.860201412619188 /loss/FBt1/assignment_loss‚0.381350392854074 1loss/SRqqt1/detection_loss‚1.9064773922336513 epoch‚4 /loss/FRt3/assignment_loss‚2.614431646562392 /loss/FRt2/detection_loss‚2.8434823671976726 1loss/SRqqt2/detection_loss‚2.5576378683890066 /loss/FRt2/assignment_loss‚4.603294618924459 /loss/FBt1/detection_loss‚2.5914423912763596ËŽˆ¤z’«‚(:‡ å¡ãÔû‘Š® gpu.0.gpu‚36.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚115.667' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1334 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0h§$2‚¬‚(:÷ å¡ãÔþ¥×± proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 366.41015625( disk.nvme0c0n1.out‚23831.79638671875) proc.memory.percent‚16.44001007080078 disk.dm-0.in‚ 366.41015625# disk.dm-0.out‚23831.79638671875& disk.nvme0n1.out‚23831.79638671875$ memory_percent‚93.75758171081543 disk.sda3.in‚ 0.0546875 disk.drbd2266.in‚0 network.sent‚ 1.3225299e+07 disk.sda.in‚ 0.0546875 disk.sda.out‚ 7586.35546875 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 366.41015625 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚13.64892578125( proc.memory.availableMB‚ 1022.7578125 cpu‚25.341646958085736 disk.drbd2266.out‚0* disk./.usagePercent‚15.133953740626438 disk.drbd2080.in‚0 disk.sda1.out‚0 proc.memory.rssMB‚ 2693.53125 disk.sda2.out‚0 disk.dm-1.in‚ 0.0546875 disk.sda3.out‚ 7586.35546875 network.recv‚ 1.630378e+06$ disk./.usageGB‚225.5068473815918 disk.drbd1716.in‚0 disk.dm-1.out‚ 7586.35546875Å6’­‚(:‡ ô¡ãÔ„œ§® gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚140.751' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.1502 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0IY€®‚(:õ ô¡ãÔˆ°Œ±) proc.memory.availableMB‚ 1022.99609375 network.sent‚ 1.3333746e+07 disk.drbd2080.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0& disk.nvme0n1.out‚23836.71044921875 disk.sda2.out‚0 disk.dm-1.in‚ 0.0546875 disk.drbd2266.out‚0# disk.dm-0.out‚23836.71044921875 disk.nvme0n1.in‚ 366.41015625 disk.sda.in‚ 0.0546875 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚23836.71044921875 disk.sda2.in‚0 disk.dm-1.out‚ 7590.6328125$ memory_percent‚93.75612735748291* disk./.usagePercent‚15.133953740626438 disk.sda3.in‚ 0.0546875 disk.sda3.out‚ 7590.6328125 disk.rbd1.out‚13.64892578125 proc.memory.rssMB‚ 2693.53125 network.recv‚ 1.637239e+06 disk.sda.out‚ 7590.6328125 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.rbd1.in‚0) proc.memory.percent‚16.44001007080078 cpu‚25.340628236540777$ disk./.usageGB‚225.5068473815918 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 366.41015625 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 366.41015625Ÿ1a󒯂(:‡ ƒ¢ãÔ—뉮 gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚119.973' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9946 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0àçŠ|°‚(:ñ ƒ¢ãÔËžÔ± disk.dm-1.out‚ 7591.1640625 cpu‚25.33965842335038* disk./.usagePercent‚15.133957580739452 disk.drbd2080.out‚0 disk.rbd1.out‚13.64892578125 disk.sda.in‚ 0.0546875 proc.memory.rssMB‚ 2693.53125 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚23839.64697265625 disk.dm-1.in‚ 0.0546875 network.sent‚ 1.3339992e+07 disk.drbd2266.in‚0 disk.sda3.out‚ 7591.1640625 disk.sda2.out‚0) proc.memory.percent‚16.44001007080078 proc.cpu.threads‚17% disk./.usageGB‚225.50690460205078 disk.rbd0.out‚0 disk.dm-0.in‚ 366.41015625# disk.dm-0.out‚23839.64697265625 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 366.41015625 disk.drbd1716.in‚0$ memory_percent‚93.75343322753906 disk.sda3.in‚ 0.0546875" disk.nvme0c0n1.in‚ 366.41015625 disk.sda.out‚ 7591.1640625% proc.memory.availableMB‚ 1023.4375 network.recv‚ 1.641835e+06 disk.drbd2265.out‚0 disk.drbd2080.in‚0& disk.nvme0n1.out‚23839.64697265625 disk.sda2.in‚0 disk.drbd1716.out‚0á|g-2±- _wandbruntime‚4503  _runtime‚4503 רeɲà _runtime‚4503.667444502 _step‚460 ' lr-AdamW/pg1‚0.00013811454831187274 ' lr-AdamW/pg2‚0.00013811454831187274 trainer/global_step‚5649 % _timestamp‚1.7883998790305996e+09÷ –Î³È % _timestamp‚1.7883998790305996e+09 _runtime‚4503.667444502 _step‚460 ' lr-AdamW/pg1‚0.00013811454831187274 ' lr-AdamW/pg2‚0.00013811454831187274 trainer/global_step‚5649Ì}åÌò2´- _wandbruntime‚4503  _runtime‚4503VA˜ ¹µÙO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114040135 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7960174892032 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚151100031646720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚166798944676864 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75343502 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚176407693705216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚5649 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74231539 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114060173 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚151100048686080 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74231539 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188291714 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚159037277853184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166798757673984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚168309469028352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158838582781952 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0z&k¦ä'¶ÞO JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 7960174892032 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚151100031646720 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚166798944676864 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75343502 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚176407693705216 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚5649 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74231539 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 114060173 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚151100048686080 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74231539 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 188291714 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚159037277853184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚166798757673984 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚168309469028352 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚158838582781952 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 114040135Í9¡uÇ2·- _wandbruntime‚4504  _runtime‚4504èU°é&¸ãM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188324186 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚159064792361036 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114079986 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75356234 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚151125974829056 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8099667897856 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32022601 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188304120 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74244198 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚151125991868416 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168338358036480 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚166827420176896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8099878469120 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚176438025934336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75355711 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74244200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7938635371008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚158865852146688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚159064610200064 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚5649 _runtime‚4504.343496721 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114059922 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚168338381976576 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74244200 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚166827607193088 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188304120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7938800492620 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7961737043456 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7961568030208 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7961737043456 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 % _timestamp‚1.7883998797060857e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚166827607193088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚158865852146688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188324186 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _step‚462 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114079986 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43333631 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚176438260445696 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43333633 SÉ븹èM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8099667897856 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32022601 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 188304120 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74244198 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚151125991868416 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚168338358036480 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚166827420176896 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8099878469120 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚176438025934336 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75355711 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74244200 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 7938635371008 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚158865852146688 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚159064610200064 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚5649 _runtime‚4504.343496721 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 114059922 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚168338381976576 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74244200 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚166827607193088 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 188304120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 7938800492620 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 7961737043456 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 7961568030208 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 7961737043456 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 % _timestamp‚1.7883998797060857e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚166827607193088 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚158865852146688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 188324186 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _step‚462 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 114079986 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚43333631 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚176438260445696 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚43333633 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 188324186 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚159064792361036 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 114079986 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75356234 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚151125974829056λU9¹2º- _wandbruntime‚4504  _runtime‚4504¹`sÅZ»Ô 1loss/SRqqt2/assignment_loss‚1.825796880669171 1loss/FBt4/assignment_loss‚0.34992866104333337 /loss/FRt3/detection_loss‚2.6447964349100666 /loss/FBt2/detection_loss‚3.2461409635014005 2loss/SRqqt3/assignment_loss‚1.5971388130428943 0loss/FBt1/assignment_loss‚0.6356873453126581 1loss/SRqqt1/detection_loss‚1.8121327126568014 .loss/FBt1/detection_loss‚2.958205020427704 .loss/FBt3/detection_loss‚2.671162210404873 trainer/global_step‚5649 0loss/FRt1/assignment_loss‚3.6087660213996626 /loss/FRt4/detection_loss‚1.9157795652242031 %loss/total_loss‚46.82357654064935 /loss/FRt2/detection_loss‚2.6426802155299063 1loss/SRqqt4/detection_loss‚1.1483661187264849 epoch‚4 _step‚463 1loss/SRqqt1/assignment_loss‚1.962736666676673 0loss/SRqqt4/assignment_loss‚1.00713524594903 1loss/FBt3/assignment_loss‚0.20831359356816392 1loss/FBt2/assignment_loss‚0.22190336730289395 1loss/SRqqt2/detection_loss‚2.6658243440812632 /loss/FBt4/detection_loss‚0.9263190938543309 $ _timestamp‚1.788399879707858e+09 0loss/FRt3/assignment_loss‚2.9615390146932294 /loss/FRt1/detection_loss‚2.0039277282254453 0loss/SRqqt3/detection_loss‚1.762213428920888 _runtime‚4504.344575557 0loss/FRt2/assignment_loss‚4.2761586384895525 0loss/FRt4/assignment_loss‚1.7709244560387174õäë…_¼Ù /loss/FRt2/detection_loss‚2.6426802155299063 1loss/SRqqt4/detection_loss‚1.1483661187264849 epoch‚4 _step‚463 1loss/SRqqt1/assignment_loss‚1.962736666676673 0loss/SRqqt4/assignment_loss‚1.00713524594903 1loss/FBt3/assignment_loss‚0.20831359356816392 1loss/FBt2/assignment_loss‚0.22190336730289395 1loss/SRqqt2/detection_loss‚2.6658243440812632 /loss/FBt4/detection_loss‚0.9263190938543309 $ _timestamp‚1.788399879707858e+09 0loss/FRt3/assignment_loss‚2.9615390146932294 /loss/FRt1/detection_loss‚2.0039277282254453 0loss/SRqqt3/detection_loss‚1.762213428920888 _runtime‚4504.344575557 0loss/FRt2/assignment_loss‚4.2761586384895525 0loss/FRt4/assignment_loss‚1.7709244560387174 1loss/SRqqt2/assignment_loss‚1.825796880669171 1loss/FBt4/assignment_loss‚0.34992866104333337 /loss/FRt3/detection_loss‚2.6447964349100666 /loss/FBt2/detection_loss‚3.2461409635014005 2loss/SRqqt3/assignment_loss‚1.5971388130428943 0loss/FBt1/assignment_loss‚0.6356873453126581 1loss/SRqqt1/detection_loss‚1.8121327126568014 .loss/FBt1/detection_loss‚2.958205020427704 .loss/FBt3/detection_loss‚2.671162210404873 trainer/global_step‚5649 0loss/FRt1/assignment_loss‚3.6087660213996626 /loss/FRt4/detection_loss‚1.9157795652242031 %loss/total_loss‚46.82357654064935Ïd8Ä ’½‚(:‡ ’¢ãÔ¼Ìݯ gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚128.353' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.6706 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0úÍA€¾‚(:õ ’¢ãÔŽ¦µ°) proc.memory.availableMB‚ 1023.41015625 network.sent‚ 1.3345385e+07 disk.drbd2266.out‚0" disk.dm-0.out‚23841.2958984375* disk./.usagePercent‚15.134310871136702 disk.sda2.in‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚23841.2958984375 proc.cpu.threads‚17% disk.nvme0n1.out‚23841.2958984375 disk.sda.in‚ 0.0546875 disk.sda3.in‚ 0.0546875$ memory_percent‚93.75360012054443 proc.memory.rssMB‚ 2693.53125 disk.dm-1.in‚ 0.0546875 disk.drbd2265.out‚0 disk.dm-0.in‚ 366.41015625 disk.sda3.out‚ 7591.70703125% disk./.usageGB‚225.51216888427734 disk.sda.out‚ 7591.70703125 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 7591.70703125 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 366.41015625) proc.memory.percent‚16.44001007080078 cpu‚25.33866064731531 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.out‚13.64892578125 network.recv‚ 1.645423e+06 disk.nvme0n1.in‚ 366.41015625—uñ ’¿‚(:‡ ¡¢ãÔáç‚® gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚129.179' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.8358 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0q²åxÀ‚(:í ¡¢ãÔ¢ôö± disk.sda3.out‚ 7594.23828125$ memory_percent‚93.75395774841309) proc.memory.percent‚16.44001007080078 disk.rbd1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 1023.3515625 disk.dm-0.in‚ 366.421875# disk.dm-0.out‚23855.02685546875 disk.rbd1.out‚14.00048828125 disk.nvme0c0n1.in‚ 366.421875 disk.drbd2266.in‚0 disk.drbd2266.out‚0 network.sent‚ 1.345372e+07 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 cpu‚25.33774751301058 disk.drbd1716.out‚0 disk.dm-1.out‚ 7594.23828125 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.0546875 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17) disk./.usagePercent‚15.13426120567506$ disk./.usageGB‚225.5114288330078 disk.sda.in‚ 0.0546875 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.0546875 proc.memory.rssMB‚ 2693.53125 network.recv‚ 1.65166e+06( disk.nvme0c0n1.out‚23855.02685546875 disk.nvme0n1.in‚ 366.421875& disk.nvme0n1.out‚23855.02685546875 disk.sda.out‚ 7594.23828125 disk.drbd2080.in‚0ýI·2Á- _wandbruntime‚4539  _runtime‚4539˜î¥ÇÂÁ & lr-AdamW/pg1‚0.0001366239158405492 & lr-AdamW/pg2‚0.0001366239158405492 trainer/global_step‚5699 % _timestamp‚1.7883999148912456e+09 _runtime‚4539.528232899 _step‚464ߢðFÌÃÆ _runtime‚4539.528232899 _step‚464 & lr-AdamW/pg1‚0.0001366239158405492 & lr-AdamW/pg2‚0.0001366239158405492 trainer/global_step‚5699 % _timestamp‚1.7883999148912456e+09ÐŒdâh2Ä- _wandbruntime‚4539  _runtime‚4539ʆß'ÅÙO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168232084236288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚160202051018752 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚160202051018752 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74864489 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75985236 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74864491 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚465 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74864489 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189917366 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177924439436800 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚160202069021696 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160404090949156 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115052875 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32290130 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8169322997248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚152397190763520 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75984676 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚169755092499456 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8006718024872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168231897233408 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8006883146276 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74864491 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8029846214656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚558 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚152397207802880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168231897233408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32289572 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚160403908788392 % _timestamp‚1.7883999148920007e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115032837 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚169755116439552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8029846214656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199­pdÍä'ÆÞO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚74864491 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚465 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚74864489 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 189917366 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚177924439436800 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚160202069021696 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚160404090949156 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 115052875 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32290130 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8169322997248 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚152397190763520 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚75984676 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚169755092499456 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8006718024872 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚168231897233408 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8006883146276 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚74864491 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8029846214656 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚558 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚152397207802880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚168231897233408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32289572 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚160403908788392 % _timestamp‚1.7883999148920007e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 115032837 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚169755116439552 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8029846214656 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚560 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚168232084236288 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚160202051018752 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚160202051018752 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚74864489 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚75985236Ñ7V½…2Ç- _wandbruntime‚4540  _runtime‚4540K5Òg ÈãM J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74877148 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚160229338386432 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚541 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 % _timestamp‚1.7883999155731285e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8031239352832 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75998139 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚160431423296244 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚152423150985216 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5699 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189929772 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74877150 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75997596 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_p)D{‚ool.allocated‚ 4624220160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚168260746752512 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115072688 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚160229320383488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚152423133945856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115052624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177954533497344 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8031408366080 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚169784029387776 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189929772 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115072688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189949838 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74877148 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74877150 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8031408366080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8008107189416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚168260559736320 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚168260746752512 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚160229320383488 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115052624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189949838 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8170528049664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 £ü%î&ÉèM NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚74877150 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8031408366080 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8008107189416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚168260559736320 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚168260746752512 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚160229320383488 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 115052624 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 189949838 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8170528049664 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚74877148 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚160229338386432 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚541 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 % _timestamp‚1.7883999155731285e+09 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8031239352832 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚75998139 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚160431423296244 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚152423150985216 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚5699 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 189929772 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚74877150 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚75997596 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚168260746752512 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 115072688 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚543 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚160229320383488 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚152423133945856 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 115052624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚177954533497344 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8031408366080 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚169784029387776 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 189929772 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 115072688 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 189949838 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚74877148 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248Òµ¢ñû2Ê- _wandbruntime‚4540  _runtime‚4540BFêZËÔ /loss/FBt3/detection_loss‚2.0871452497584477 0loss/SRqqt3/assignment_loss‚1.36451576771931 /loss/FRt1/detection_loss‚2.0886239403603124 _step‚467 1loss/SRqqt2/assignment_loss‚2.276772981095645 1loss/SRqqt1/assignment_loss‚1.522563307486633 0loss/FBt2/assignment_loss‚0.5672950313704246 .loss/FRt2/detection_loss‚3.028046869314634 1loss/SRqqt3/detection_loss‚1.7637754148588731 .loss/FBt2/detection_loss‚2.594718486070633 epoch‚4 2loss/SRqqt4/assignment_loss‚1.1079445508948993 /loss/FBt1/detection_loss‚3.0685190558433533 %loss/total_loss‚47.18345418919262 _runtime‚4540.211511846 0loss/FRt1/assignment_loss‚3.4119903597177244 1loss/SRqqt1/detection_loss‚1.7779033650109108 1loss/SRqqt4/detection_loss‚1.4084167366381735 /loss/FBt4/detection_loss‚0.9613158139400184 % _timestamp‚1.7883999155747674e+09 0loss/FRt4/assignment_loss‚1.9634660932252992 1loss/FBt1/assignment_loss‚0.22717459059731482 0loss/FBt3/assignment_loss‚1.0281205827403548 /loss/FRt4/detection_loss‚1.8915242648349617 1loss/SRqqt2/detection_loss‚2.4359157896704144 trainer/global_step‚5699 /loss/FRt2/assignment_loss‚4.567963792727544 0loss/FRt3/assignment_loss‚3.5602828593964273 1loss/FBt4/assignment_loss‚0.19181979133127242 /loss/FRt3/detection_loss‚2.2876394945890346Q!9ü_ÌÙ 1loss/SRqqt2/assignment_loss‚2.276772981095645 1loss/SRqqt1/assignment_loss‚1.522563307486633 0loss/FBt2/assignment_loss‚0.5672950313704246 .loss/FRt2/detection_loss‚3.028046869314634 1loss/SRqqt3/detection_loss‚1.7637754148588731 .loss/FBt2/detection_loss‚2.594718486070633 epoch‚4 2loss/SRqqt4/assignment_loss‚1.1079445508948993 /loss/FBt1/detection_loss‚3.0685190558433533 %loss/total_loss‚47.18345418919262 _runtime‚4540.211511846 0loss/FRt1/assignment_loss‚3.4119903597177244 1loss/SRqqt1/detection_loss‚1.7779033650109108 1loss/SRqqt4/detection_loss‚1.4084167366381735 /loss/FBt4/detection_loss‚0.9613158139400184 % _timestamp‚1.7883999155747674e+09 0loss/FRt4/assignment_loss‚1.9634660932252992 1loss/FBt1/assignment_loss‚0.22717459059731482 0loss/FBt3/assignment_loss‚1.0281205827403548 /loss/FRt4/detection_loss‚1.8915242648349617 1loss/SRqqt2/detection_loss‚2.4359157896704144 trainer/global_step‚5699 /loss/FRt2/assignment_loss‚4.567963792727544 0loss/FRt3/assignment_loss‚3.5602828593964273 1loss/FBt4/assignment_loss‚0.19181979133127242 /loss/FRt3/detection_loss‚2.2876394945890346 /loss/FBt3/detection_loss‚2.0871452497584477 0loss/SRqqt3/assignment_loss‚1.36451576771931 /loss/FRt1/detection_loss‚2.0886239403603124 _step‚467Ó•¡ÂÖ‘Í‚(:† °¢ãÔÛß® gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚132.667' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.5334 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08j¾t΂(:é °¢ãÔôËœ± disk.sda1.in‚0 disk.nvme0c0n1.in‚ 366.421875 disk.dm-1.out‚ 7595.19140625) proc.memory.availableMB‚ 1023.80859375 network.sent‚ 1.3459615e+07 disk.sda1.out‚0 disk.rbd1.out‚14.07861328125 disk.sda.in‚ 0.0546875 disk.sda.out‚ 7595.19140625 disk.drbd2080.out‚0 network.recv‚ 1.655633e+06 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 366.421875 disk.drbd2266.in‚0 disk.rbd0.out‚0) proc.memory.percent‚16.44001007080078 disk.rbd1.in‚0 disk.dm-1.in‚ 0.0546875 disk.drbd2080.in‚0! disk.dm-0.out‚23855.302734375 proc.cpu.threads‚17) disk./.usagePercent‚15.13426120567506 disk.sda3.in‚ 0.0546875 disk.sda2.out‚0 disk.drbd1716.in‚0# memory_percent‚93.7511682510376$ disk.nvme0n1.out‚23855.302734375 disk.dm-0.in‚ 366.421875 proc.memory.rssMB‚ 2693.53125 cpu‚25.33681120050814$ disk./.usageGB‚225.5114288330078 disk.sda3.out‚ 7595.19140625& disk.nvme0c0n1.out‚23855.302734375 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0Ic&ûÏ‚(:’ ¿¢ãÔìç ® gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚136.974' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.394799999999996 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0çó.gЂ(:Ü ¿¢ãԪǰ disk.rbd0.in‚0 disk.sda1.in‚0 cpu‚25.335880419442937 disk.rbd1.out‚14.07861328125 disk.sda.in‚ 0.0546875 disk.sda2.out‚0( proc.memory.availableMB‚ 1023.5546875 proc.memory.rssMB‚ 2693.53125) proc.memory.percent‚16.44001007080078 disk.sda.out‚7595.625 disk.rbd0.out‚0 network.sent‚ 1.3568386e+07! disk.dm-0.out‚23857.720703125 disk.drbd2080.out‚0 disk.sda3.out‚7595.625 disk.rbd1.in‚0 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.dm-0.in‚ 366.421875 disk.nvme0c0n1.in‚ 366.421875 disk.drbd1716.out‚0$ memory_percent‚93.75271797180176 disk.dm-1.in‚ 0.0546875 disk.sda3.in‚ 0.0546875 disk.nvme0n1.in‚ 366.421875$ disk.nvme0n1.out‚23857.720703125 network.recv‚ 1.662336e+06 disk.dm-1.out‚7595.625 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.drbd1716.in‚0) disk./.usagePercent‚15.13422741268054% disk./.usageGB‚225.51092529296875& disk.nvme0c0n1.out‚23857.720703125 proc.cpu.threads‚17‹RAÑ‚(:’ ΢ãÔî’Š® gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚140.221' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.044200000000004 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0[Ó9mwÒ‚(:ì ΢ãÔœ¦ç° network.sent‚ 1.3573705e+07* disk./.usagePercent‚15.134222292529856 disk.rbd1.in‚0 disk.rbd1.out‚14.11083984375 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.memory.rssMB‚ 2693.53125) proc.memory.percent‚16.44001007080078 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.0546875 disk.drbd2266.out‚0! disk.dm-0.out‚23857.962890625 disk.sda1.in‚0 disk.nvme0n1.in‚ 366.421875( proc.memory.availableMB‚ 1023.5546875 disk.drbd2266.in‚0& disk.nvme0c0n1.out‚23857.962890625% disk./.usageGB‚225.51084899902344 disk.drbd2265.in‚0 disk.dm-0.in‚ 366.421875$ disk.nvme0n1.out‚23857.962890625 disk.sda.in‚ 0.0546875 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.sda3.out‚ 7596.29296875 disk.sda.out‚ 7596.29296875 cpu‚25.334890952453613 disk.dm-1.out‚ 7596.29296875 disk.sda1.out‚0 disk.sda3.in‚ 0.0546875$ memory_percent‚93.75271797180176 proc.cpu.threads‚17 network.recv‚ 1.665754e+06 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.nvme0c0n1.in‚ 366.421875uÉ"F2Ó- _wandbruntime‚4575  _runtime‚4575‚>Ý!ÇÔÁ _runtime‚4575.319288176 _step‚468 & lr-AdamW/pg1‚0.0001351272125939678 & lr-AdamW/pg2‚0.0001351272125939678 trainer/global_step‚5749 % _timestamp‚1.7883999506824322e+09û7‰ÌÕÆ & lr-AdamW/pg1‚0.0001351272125939678 & lr-AdamW/pg2‚0.0001351272125939678 trainer/global_step‚5749 % _timestamp‚1.7883999506824322e+09 _runtime‚4575.319288176 _step‚468ÔéP‰™2Ö- _wandbruntime‚4575  _runtime‚4575g?:µ!×ÙO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44063956 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44063954 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32562193 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8239970301440 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚153694349880320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 % _timestamp‚1.7883999506831343e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191543018 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32561663 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76626149 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116025539 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚5749 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚179440710212096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚169665223795712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75497441 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76625617 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8240187177472 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚161565537258496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚161565519255552 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚161565519255552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚161565537258496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚169665036792832 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚171200739910656 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚179440951028224 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚171200763850752 _runtime‚4575.320271151 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚161770721884364TËA¨ä'ØÞO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44063956 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44063954 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚32562193 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8239970301440 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚153694349880320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 % _timestamp‚1.7883999506831343e+09 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 191543018 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32561663 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚76626149 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 116025539 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚5749 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚179440710212096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚169665223795712 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚75497441 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚76625617 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8240187177472 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚161565537258496 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚161565519255552 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚161565519255552 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚161565537258496 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚169665036792832 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚171200739910656 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚179440951028224 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚171200763850752 _runtime‚4575.320271151 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚161770721884364Õ|à¤2Ù- _wandbruntime‚4576  _runtime‚4576‘Èêé&ÚãM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8101079688704 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8100910675456 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 trainer/global_step‚5749 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8101079688704 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191575490 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32567005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚169693886311936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191555424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75510100 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44071331 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116045326 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76638869 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32567536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚4576.221431016 _step‚470 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚169693699295744 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8077744129436 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8241600775168 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚171229676798976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8241381815296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚153720293062656 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116045326 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8077579007824 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7883999515839133e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75510098 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191575490 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75510100 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚161592806623232 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116065390 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚533 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8100910675456 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75510098 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179471034674176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚161798054231452 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116065390 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚161592788620288 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚161797872070480 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44071333 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚161592788620288 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76638336 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408À®‰È &ÛèM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚179471277574144 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚171229652858880 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚75510100 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚161592806623232 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 116065390 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚533 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8100910675456 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚75510098 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚179471034674176 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚161798054231452 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 116065390 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚161592788620288 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚161797872070480 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44071333 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚161592788620288 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚76638336 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8101079688704 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8100910675456 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 trainer/global_step‚5749 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚531 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8101079688704 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 191575490 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32567005 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚169693886311936 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 191555424 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚75510100 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44071331 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 116045326 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚76638869 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚32567536 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 _runtime‚4576.221431016 _step‚470 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚169693699295744 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8077744129436 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8241600775168 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚171229676798976 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8241381815296 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚153720293062656 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 116045326 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8077579007824 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7883999515839133e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚75510098 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 191575490 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 EÎtor.on_train_batch_end/allocation.all.current‚20066 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217ÖàyË2Ü- _wandbruntime‚4576  _runtime‚4576@Ô‡[WÝÑ 0loss/FBt1/assignment_loss‚0.6102565195983166 _step‚471 0loss/FRt1/assignment_loss‚3.5262375438213347 0loss/SRqqt1/detection_loss‚1.760755183137193 epoch‚4 % _timestamp‚1.7883999515859792e+09 2loss/SRqqt1/assignment_loss‚1.5938115314561494 1loss/FBt4/assignment_loss‚0.21291188820337706 .loss/FRt4/detection_loss‚2.752786728643602 /loss/FBt4/detection_loss‚0.6218716521899021 0loss/FRt4/assignment_loss‚1.6944319971146122 1loss/SRqqt2/assignment_loss‚2.028739173051923 1loss/FBt3/assignment_loss‚0.47660640859792885 1loss/SRqqt2/detection_loss‚2.3360580885133078 .loss/FRt2/detection_loss‚2.644347811854163 1loss/SRqqt3/detection_loss‚1.8756097455819447 1loss/SRqqt4/detection_loss‚1.2129399058447308 1loss/FBt2/assignment_loss‚0.37399787100201304 .loss/FBt2/detection_loss‚2.627588475292379 -loss/FBt3/detection_loss‚3.37040046283177 trainer/global_step‚5749 /loss/FRt2/assignment_loss‚3.649873899859051 2loss/SRqqt3/assignment_loss‚1.4883143945651895 /loss/FRt1/detection_loss‚2.2865873005986215 .loss/FRt3/detection_loss‚2.745420358247227 .loss/FBt1/detection_loss‚2.095017304787269 %loss/total_loss‚46.59259788505221 _runtime‚4576.222703316 /loss/FRt3/assignment_loss‚3.337076192100843 2loss/SRqqt4/assignment_loss‚1.2709574481593573fï5º\ÞÖ 1loss/SRqqt2/assignment_loss‚2.028739173051923 1loss/FBt3/assignment_loss‚0.47660640859792885 1loss/SRqqt2/detection_loss‚2.3360580885133078 .loss/FRt2/detection_loss‚2.644347811854163 1loss/SRqqt3/detection_loss‚1.8756097455819447 1loss/SRqqt4/detection_loss‚1.2129399058447308 1loss/FBt2/assignment_loss‚0.37399787100201304 .loss/FBt2/detection_loss‚2.627588475292379 -loss/FBt3/detection_loss‚3.37040046283177 trainer/global_step‚5749 /loss/FRt2/assignment_loss‚3.649873899859051 2loss/SRqqt3/assignment_loss‚1.4883143945651895 /loss/FRt1/detection_loss‚2.2865873005986215 .loss/FRt3/detection_loss‚2.745420358247227 .loss/FBt1/detection_loss‚2.095017304787269 %loss/total_loss‚46.59259788505221 _runtime‚4576.222703316 /loss/FRt3/assignment_loss‚3.337076192100843 2loss/SRqqt4/assignment_loss‚1.2709574481593573 0loss/FBt1/assignment_loss‚0.6102565195983166 _step‚471 0loss/FRt1/assignment_loss‚3.5262375438213347 0loss/SRqqt1/detection_loss‚1.760755183137193 epoch‚4 % _timestamp‚1.7883999515859792e+09 2loss/SRqqt1/assignment_loss‚1.5938115314561494 1loss/FBt4/assignment_loss‚0.21291188820337706 .loss/FRt4/detection_loss‚2.752786728643602 /loss/FBt4/detection_loss‚0.6218716521899021 0loss/FRt4/assignment_loss‚1.6944319971146122×o§t’ß‚(:‡ Ý¢ãÔ‹¤ª® gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚143.173' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.6346 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£; |à‚(:ñ Ý¢ãÔã¯ê° proc.memory.rssMB‚ 2693.53125$ disk./.usageGB‚226.8418426513672 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 366.69140625 cpu‚25.33396472478665 disk.sda3.out‚ 7596.8671875 disk.rbd1.in‚0 disk.sda2.out‚0 disk.drbd2080.in‚0) proc.memory.percent‚16.44001007080078 disk.dm-1.in‚ 0.0546875& disk.nvme0n1.out‚23858.98486328125 network.recv‚ 1.670244e+06 disk.sda1.in‚0 disk.drbd1716.in‚0# disk.dm-0.out‚23858.98486328125( disk.nvme0c0n1.out‚23858.98486328125 disk.nvme0n1.in‚ 366.69140625 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 366.69140625 disk.sda.out‚ 7596.8671875# memory_percent‚93.7565803527832 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 1022.921875 disk.dm-1.out‚ 7596.8671875 disk.drbd2265.out‚0 disk.drbd2080.out‚0 network.sent‚ 1.3579724e+07* disk./.usagePercent‚15.223546393316703 disk.sda3.in‚ 0.0546875 disk.rbd1.out‚14.11083984375 disk.sda.in‚ 0.0546875 disk.sda2.in‚0]ÄÑá‚(:’ ì¢ãÔÊ憮 gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚150.734' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.146800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Êf»â‚(:ô ì¢ãÔ±Ôƒ± disk.sda.out‚ 7597.62109375 disk.drbd1716.out‚0 disk.drbd2265.in‚0 network.recv‚ 1.676566e+06* disk./.usagePercent‚15.223227151921506( disk.nvme0c0n1.out‚25217.79736328125 disk.drbd2080.out‚0 disk.rbd0.in‚0) proc.memory.percent‚16.44001007080078% disk./.usageGB‚226.83708572387695 disk.drbd2266.in‚0' proc.memory.availableMB‚ 1022.828125 proc.cpu.threads‚17& disk.nvme0n1.out‚25217.79736328125 disk.drbd1716.in‚0 disk.dm-1.out‚ 7597.62109375 network.sent‚ 1.368799e+07 disk.nvme0n1.in‚ 370.30859375 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 7597.62109375 disk.sda1.in‚0 disk.rbd1.out‚ 14.173828125 proc.memory.rssMB‚ 2693.53125 cpu‚25.333042616389527" disk.nvme0c0n1.in‚ 370.30859375 disk.dm-1.in‚ 0.0546875 disk.dm-0.in‚ 370.30859375 disk.sda1.out‚0 disk.sda3.in‚ 0.0546875$ memory_percent‚93.75715255737305 disk.sda.in‚ 0.0546875 disk.drbd2266.out‚0# disk.dm-0.out‚25217.79736328125 disk.rbd1.in‚0ª5Jæ2ã- _wandbruntime‚4611  _runtime‚4611·!VÇäÁ & lr-AdamW/pg1‚0.0001336247504828418 & lr-AdamW/pg2‚0.0001336247504828418 trainer/global_step‚5799 % _timestamp‚1.7883999865316393e+09 _runtime‚4611.168525956 _step‚472bœÔzÌåÆ % _timestamp‚1.7883999865316393e+09 _runtime‚4611.168525956 _step‚472 & lr-AdamW/pg1‚0.0001336247504828418 & lr-AdamW/pg2‚0.0001336247504828418 trainer/global_step‚5799Ø6¬á92æ- _wandbruntime‚4611  _runtime‚4611‚ Àß'çÙO JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚163137170658808 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44432806 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚172646387321856 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77268108 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193148630 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117018241 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8169357859840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162929005495296 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 % _timestamp‚1.7883999865323496e+09 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117038279 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32835304 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚154991508997120 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8145661661688 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚180957240532992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117038279 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚5799 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚171098176352256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76130391 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76130389 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚171098363355136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8310853211136 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚154991526036480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚171098176352256 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚473 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚549 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162929005495296 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8169357859840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77268657 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚180957475057664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76130391 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8169188859904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76130389 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172646411261952 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 _runtime‚4611.169437246 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚171098363355136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8169188859904 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26JDwèä'èÞO UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 117038279 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚5799 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚171098176352256 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚76130391 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚76130389 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚171098363355136 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8310853211136 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚154991526036480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚171098176352256 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚473 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚549 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚162929005495296 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8169357859840 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77268657 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚180957475057664 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76130391 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚547 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8169188859904 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76130389 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚172646411261952 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 _runtime‚4611.169437246 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚171098363355136 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8169188859904 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚163137170658808 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44432806 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚172646387321856 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚77268108 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 193148630 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 117018241 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8169357859840 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚162929005495296 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 % _timestamp‚1.7883999865323496e+09 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 117038279 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚32835304 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚154991508997120 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8145661661688 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚180957240532992 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160ÙÓl‚2é- _wandbruntime‚4611  _runtime‚4611]”áÄÇêãM RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚163164685166660 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8170581998080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚180987574481408 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚155017469218816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193201142 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162956256857088 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162956256857088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚163164503005688 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8147050826232 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76143050 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44440183 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193181076 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚171126838855168 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117038028 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117058092 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32840627 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8170751011328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44440181 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77280808 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8312482685440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8147215947844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162956274860032 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76143048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193201142 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76143050 % _timestamp‚1.7883999871939528e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚171127025871360 _step‚474 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚171127025871360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8170751011328 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117038028oÆK§î&ëèM E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8170581998080 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚180987574481408 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚155017469218816 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 193201142 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚162956256857088 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚162956256857088 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚539 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚163164503005688 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8147050826232 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚76143050 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44440183 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 193181076 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚171126838855168 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 117038028 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 117058092 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚32840627 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8170751011328 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44440181 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77280808 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8312482685440 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8147215947844 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚162956274860032 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76143048 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 193201142 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76143050 % _timestamp‚1.7883999871939528e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚171127025871360 _step‚474 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚171127025871360 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8170751011328 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 117038028 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚163164685166660 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 Fü6Åô¿ & lr-AdamW/pg2‚0.0001321168426180232 trainer/global_step‚5849 # _timestamp‚1.78840002250576e+09 _runtime‚4647.142632741 _step‚476 & lr-AdamW/pg1‚0.0001321168426180232µpÊõÄ & lr-AdamW/pg1‚0.0001321168426180232 & lr-AdamW/pg2‚0.0001321168426180232 trainer/global_step‚5849 # _timestamp‚1.78840002250576e+09 _runtime‚4647.142632741 _step‚476ÜË;J<2ö- _wandbruntime‚4647  _runtime‚4647|ÐÈÞ'÷ØO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76763339 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚477 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77910178 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚535 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8381769769984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8215298601500 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76763341 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚172531315911680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8215133480096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44801654 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚164503801594016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚172531502914560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8239029182464 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44801656 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚156288685153280 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172531315911680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118030981 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚182473804503040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0¬/ ­øÝO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚164292473732096 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8238860182528 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚174092058673152 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚182473804503040 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚76763339 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚477 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚77910178 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚535 _MDeviceStatsMonitor.on_traiàš‹¹6!n_batch_start/inactive_split_bytes.small_pool.freed‚ 8381769769984 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8215298601500 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚76763341 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚172531315911680 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8215133480096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚44801654 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚164503801594016 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚172531502914560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8239029182464 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚44801656 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚156288685153280 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚172531315911680 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 118030981 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚164319743096832 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8216522644640 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 _step‚478 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76775998 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118030730 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33113463 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚172560165430784 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44809033 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚156314611296256 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8216687766252 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚172560165430784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚164319743096832 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8240422333952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚182504138750464 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8240253320704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚164319725093888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚164531133940896 _runtime‚4647.832560335 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194826794 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚164531316101868 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76776000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194826794 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118050794 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8240422333952 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76775998 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77922494 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44809031 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8240253320704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174120947681280 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33114000 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚182504369067520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118050794 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8383397446144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0Ä…nÏî&ûèM \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8240422333952 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚182504138750464 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8240253320704 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚164319725093888 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚164531133940896 _runtime‚4647.832560335 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 194826794 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚164531316101868 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚76776000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 194826794 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 118050794 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8240422333952 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚76775998 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚77922494 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚44809031 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8240253320704 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚174120947681280 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33114000 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚182504369067520 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 118050794 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8383397446144 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 118030730 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚164319743096832 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8216522644640 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 _step‚478 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚76775998 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 118030730 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33113463 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚172560165430784 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚44809033 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚156314611296256 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8216687766252 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚172560165430784 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚164319743096832Þ²lX2ü- _wandbruntime‚4647  _runtime‚4647àGmZýÔ 0loss/FRt4/assignment_loss‚1.3475102759757132 1loss/SRqqt4/detection_loss‚1.3723946375805034 trainer/global_step‚5849 /loss/FRt3/assignment_loss‚3.403415396099999 1loss/FBt4/assignment_loss‚0.21906201161710367 /loss/FRt2/detection_loss‚3.1760023729554536 /loss/FBt2/detection_loss‚2.4720546044409275 .loss/FBt3/detection_loss‚2.400746151804924 /loss/FBt4/detection_loss‚0.5964997789955565 _runtime‚4647.833526749 _step‚479 epoch‚4 % _timestamp‚1.7884000231967914e+09 2loss/SRqqt4/assignment_loss‚0.9343919859578212 0loss/FBt3/assignment_loss‚0.4142412942122367 .loss/FRt1/detection_loss‚2.033251253101563 2loss/SRqqt1/assignment_loss‚1.7534674752284498 2loss/SRqqt3/assignment_loss‚1.4403650117633136 1loss/SRqqt2/assignment_loss‚1.790088207593986 0loss/FBt1/assignment_loss‚0.6142267243288059 /loss/FRt4/detection_loss‚1.5849485258167644 &loss/total_loss‚44.586886316297495 /loss/FRt1/assignment_loss‚3.725436340437995 0loss/FBt2/assignment_loss‚0.5525687709741192 /loss/FRt3/detection_loss‚2.5633055177472888 1loss/SRqqt1/detection_loss‚1.9800017937725665 0loss/SRqqt2/detection_loss‚2.678697177342006 1loss/SRqqt3/detection_loss‚1.8987755224756573 .loss/FBt1/detection_loss‚2.417062623160226 0loss/FRt2/assignment_loss‚3.2183728629145127‡¼{_þÙ .loss/FBt1/detection_loss‚2.417062623160226 0loss/FRt2/assignment_loss‚3.2183728629145127 0loss/FRt4/assignment_loss‚1.3475102759757132 1loss/SRqqt4/detection_loss‚1.3723946375805034 trainer/global_step‚5849 /loss/FRt3/assignment_loss‚3.403415396099999 1loss/FBt4/assignment_loss‚0.21906201161710367 /loss/FRt2/detection_loss‚3.1760023729554536 /loss/FBt2/detection_loss‚2.4720546044409275 .loss/FBt3/detection_loss‚2.400746151804924 /loss/FBt4/detection_loss‚0.5964997789955565 _runtime‚4647.833526749 _step‚479 epoch‚4 % _timestamp‚1.7884000231967914e+09 2loss/SRqqt4/assignment_loss‚0.9343919859578212 0loss/FBt3/assignment_loss‚0.4142412942122367 .loss/FRt1/detection_loss‚2.033251253101563 2loss/SRqqt1/assignment_loss‚1.7534674752284498 2loss/SRqqt3/assignment_loss‚1.4403650117633136 1loss/SRqqt2/assignment_loss‚1.790088207593986 0loss/FBt1/assignment_loss‚0.6142267243288059 /loss/FRt4/detection_loss‚1.5849485258167644 &loss/total_loss‚44.586886316297495 /loss/FRt1/assignment_loss‚3.725436340437995 0loss/FBt2/assignment_loss‚0.5525687709741192 /loss/FRt3/detection_loss‚2.5633055177472888 1loss/SRqqt1/detection_loss‚1.9800017937725665 0loss/SRqqt2/detection_loss‚2.678697177342006 1loss/SRqqt3/detection_loss‚1.8987755224756573ß?.Iÿ‚(:’ ™£ãÔÜ®ž° gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚136.974' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.394799999999996 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Uøú‚ú€‚(:÷ ™£ãÔÑଲ disk.rbd0.out‚0 disk.dm-0.in‚ 370.58984375 disk.sda3.out‚ 7599.01953125 disk.sda2.out‚0 proc.cpu.threads‚17 disk.rbd0.in‚0* disk./.usagePercent‚15.223189006798904% disk./.usageGB‚226.83651733398438 disk.drbd2265.in‚0 disk.dm-1.out‚ 7599.01953125$ memory_percent‚93.75712871551514 disk.drbd2080.out‚0# disk.dm-0.out‚25220.01318359375& disk.nvme0n1.out‚25220.01318359375 disk.sda.in‚ 0.0546875 proc.memory.rssMB‚ 2693.53125 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.0546875 disk.sda2.in‚0 disk.drbd1716.out‚0) proc.memory.percent‚16.44001007080078) proc.memory.availableMB‚ 1022.83203125 cpu‚25.330249655775013 network.sent‚ 1.3808048e+07 network.recv‚ 1.691003e+06 disk.sda1.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 14.236328125( disk.nvme0c0n1.out‚25220.01318359375" disk.nvme0c0n1.in‚ 370.58984375 disk.nvme0n1.in‚ 370.58984375 disk.sda.out‚ 759‡Þ44ˆ9.01953125 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.0546875 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0Ãê)‚(:’ ¨£ãÔ¬ü­ gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚151.793' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.358600000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ß<*wm‚‚(:â ¨£ãÔºÿë¯ disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.sda2.in‚0 network.sent‚ 1.391632e+07 disk.dm-1.out‚7599.625 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 370.58984375' disk.nvme0c0n1.out‚25220.5146484375 disk.drbd1716.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.0546875 disk.sda3.out‚7599.625) proc.memory.availableMB‚ 1022.73828125 proc.memory.rssMB‚ 2693.53125 disk.sda.out‚7599.625 disk.drbd2080.out‚0 disk.dm-0.in‚ 370.58984375 disk.rbd1.out‚ 14.251953125% disk.nvme0n1.out‚25220.5146484375) proc.memory.percent‚16.44001007080078 disk.dm-1.in‚ 0.0546875 disk.drbd2265.in‚0 disk.rbd0.out‚0 network.recv‚ 1.696937e+06 disk.rbd1.in‚0 disk.nvme0n1.in‚ 370.58984375 disk.sda.in‚ 0.0546875 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0$ memory_percent‚93.75770092010498 proc.cpu.threads‚17* disk./.usagePercent‚15.223190286836576$ disk./.usageGB‚226.8365364074707" disk.dm-0.out‚25220.5146484375 disk.sda1.out‚0 cpu‚25.32938150560834›ö-ú’ƒ‚(:‡ ·£ãԯЙ® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚126.172' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.2344 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)Ä‚(:ö ·£ãÔßüö±* disk./.usagePercent‚15.223304210189312 disk.sda1.in‚0 disk.sda.out‚ 7600.11328125 disk.drbd1716.out‚0) proc.memory.percent‚16.44001007080078 disk.rbd0.out‚0 disk.dm-0.in‚ 370.58984375 disk.sda3.out‚ 7600.11328125" disk.nvme0c0n1.in‚ 370.58984375 proc.cpu.threads‚17 network.recv‚ 1.700883e+06$ disk./.usageGB‚226.8382339477539 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0( disk.nvme0c0n1.out‚25223.27490234375 disk.sda.in‚ 0.0546875$ memory_percent‚93.75770092010498 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.rbd1.out‚ 14.251953125& disk.nvme0n1.out‚25223.27490234375 disk.dm-1.in‚ 0.0546875 disk.dm-1.out‚ 7600.11328125) proc.memory.availableMB‚ 1022.73828125 cpu‚25.328467457475384# disk.dm-0.out‚25223.27490234375 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.nvme0n1.in‚ 370.58984375 network.sent‚ 1.3921813e+07 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.0546875 disk.drbd2080.in‚0 disk.rbd1.in‚0 proc.memory.rssMB‚ 2693.53125{‚2…- _wandbruntime‚4682  _runtime‚46823’JdžÁ & lr-AdamW/pg1‚0.0001306038032452509 & lr-AdamW/pg2‚0.0001306038032452509 trainer/global_step‚5899 % _timestamp‚1.7884000580809078e+09 _runtime‚4682.717787146 _step‚480˜€Š?Ì‡Æ _step‚480 & lr-AdamW/pg1‚0.0001306038032452509 & lr-AdamW/pg2‚0.0001306038032452509 trainer/global_step‚5899 % _timestamp‚1.7884000580809078e+09 _runtime‚4682.717787146àùòÎó2ˆ- _wandbruntime‚4682  _runtime‚4682åŒ=êÞ'‰ØO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45170504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8452589014528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚165870432529224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚165655923965952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚173964455471104 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78551540 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 _step‚481 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚165655941968896 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 _runtime‚4682.718731699 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8308700505088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚183990271158784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77396289 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8284770419908 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚173964455471104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77396291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33381034 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119003645 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196419974 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78550990 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 196399934 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119003645 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45170506 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8284605298504 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119023683 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8452801696256 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 trainer/global_step‚5899 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77396291 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8308531505152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚173964642473984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚175537682144256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33380486 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚165870614689988 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157585827230720 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217xH„ã'ŠÝO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚77396289 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8284770419908 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚173964455471104 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚77396291 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33381034 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119003645 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 196419974 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚78550990 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 196399934 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 119003645 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45170506 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8284605298504 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 119023683 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8452801696256 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 trainer/global_step‚5899 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚77396291 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8308531505152 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚173964642473984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚175537682144256 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚33380486 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚165870614689988 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚157585827230720 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45170504 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8452589014528 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚165870432529224 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚165655923965952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚173964455471104 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚78551540 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 _step‚481 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚165655941968896 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 _runtime‚4682.718731699 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8308700505088 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚183990271158784 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944á]Ž7«2‹- _wandbruntime‚4683  _runtime‚4683ôíÁ×!ŒãM L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78563719 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196452446 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33385838 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196432380 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8309924643328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚173993117974016 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚533 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119023432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚165683193330688 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78564254 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8310093656576 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8310093656576 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8454005726208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196452446 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8309924643328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184020837427200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196432380 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8285994463048 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚173993304990208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33386371 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚175566595092480 _runtime‚4683.393471447 _step‚482 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119043496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚165897947037076>õpâî&èM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚77408948 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚535 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚165683211333632 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚77408950 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45177883 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 119023432 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45177881 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚77408950 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚165683193330688 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚77408948 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚165897764876104 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 196432380 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8285994463048 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚173993304990208 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33386371 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚175566595092480 _runtime‚4683.393471447 _step‚482 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 119043496 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚165897947037076 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚78563719 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 196452446 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33385838 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 196432380 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8309924643328 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚173993117974016 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚533 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 119023432 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚165683193330688 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚78564254 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8310093656576 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8310093656576 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8454005726208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 196452446 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8309924643328 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚184020837427200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚158883003386880 trainer/global_step‚5949 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45539356 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚167019410205696 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚158882986347520 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78029241 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚176983353495552 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8354077116912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8378371827712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78029239 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚176983329555456 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8523659948032 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120016385 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚167237063464432 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚175397595030528 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚175397782033408 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8378202827776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79192524 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚4718.611600849 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78029241 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 _step‚485 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚175397595030528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198045626 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚167019392202752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8354242238316 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884000939745963e+098R0QÝšÞO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78029239 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚176983329555456 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8523659948032 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 120016385 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚167237063464432 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚175397595030528 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚175397782033408 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8378202827776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚79192524 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚4718.611600849 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚78029241 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 _step‚485 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚175397595030528 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 198045626 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚167019392202752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8354242238316 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884000939745963e+09 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 120016385 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚45539354 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 119996347 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚175397782033408 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8523449363456 UGDeviceStatsMonitor.on_train_batch_´ö®start/active_bytes.small_pool.current‚ 168999936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 198025586 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚167019410205696 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚167019392202752 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78029239 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚185507013443584 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚185506778918912 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8378202827776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚566 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚167237245625196 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33653736 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚158883003386880 trainer/global_step‚5949 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45539356 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚167019410205696 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚158882986347520 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78029241 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚176983353495552 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8354077116912 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8378371827712 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2åV$Hù2›- _wandbruntime‚4719  _runtime‚4719ÛiN´é&œãM REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78041900 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8355631403068 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198058032 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78041898 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158908946569216 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8355466281456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45546731 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45546733 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚167264577972284 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚167046679570432 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚175426257533440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚185537345431040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8524866318848 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120016134 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33658511 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚185537108822528 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚158908929529856 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79205786 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79205242 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8525078987264 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8379764979200 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198078098 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198078098 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120036198 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚167264395811312 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚177012266443776 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _runtime‚4719.285724907 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚167046661567488 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120036198 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120016134 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78041900 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚167046679570432 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8379764979200 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚175426444549632 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78041898 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33659053 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚177012242503680 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 _step‚486 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚167046661567488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 % _timestamp‚1.7884000946484838e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064\Êî&èM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚177012266443776 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _runtime‚4719.285724907 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚167046661567488 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 120036198 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 120016134 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚78041900 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚167046679570432 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8379764979200 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚175426444549632 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78041898 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚33659053 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚177012242503680 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 _step‚486 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚167046661567488 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 % _timestamp‚1.7884000946484838e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚78041900 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8355631403068 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 198058032 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78041898 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚158908946569216 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8355466281456 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45546731 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45546733 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚167264577972284 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚167046679570432 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚175426257533440 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚544 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚185537345431040 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8524866318848 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 120016134 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚33658511 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚185537108822528 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚158908929529856 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79205786 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79205242 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8525078987264 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8379764979200 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 198078098 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 198078098 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 120036198 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚167264395811312 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0æʽã&2ž- _wandbruntime‚4719  _runtime‚4719õ Ô¹ZŸÔ 1loss/FBt1/assignment_loss‚0.48810924325759214 .loss/FBt2/detection_loss‚2.623604279298049 0loss/FRt4/assignment_loss‚1.6844674562508206 2loss/SRqqt3/assignment_loss‚1.2396324376630432 1loss/FBt2/assignment_loss‚0.41430356763435855 1loss/SRqqt3/detection_loss‚1.7099792413851793 # _timestamp‚1.78840009465004e+09 /loss/FRt3/assignment_loss‚3.164890603963719 0loss/SRqqt4/detection_loss‚1.222224473146101 /loss/FBt4/detection_loss‚0.6505940780043602 epoch‚4 2loss/SRqqt2/assignment_loss‚1.6369057452237163 1loss/FBt3/assignment_loss‚0.27181429635735793 .loss/FBt3/detection_loss‚3.495490332444509 2loss/SRqqt1/assignment_loss‚1.3754775579353529 /loss/FRt3/detection_loss‚2.5320963707081106 1loss/SRqqt1/detection_loss‚1.8214756729347366 &loss/total_loss‚45.448155417716926 _step‚487 -loss/FRt4/detection_loss‚1.77002843764593 1loss/SRqqt2/detection_loss‚2.6019603510697684 2loss/SRqqt4/assignment_loss‚0.8423101702705026 1loss/FBt4/assignment_loss‚0.15100753534511507 /loss/FRt2/detection_loss‚3.3791981261709463 trainer/global_step‚5949 _runtime‚4719.286760612 0loss/FRt1/assignment_loss‚3.0274462894751477 /loss/FRt2/assignment_loss‚4.442247141962466 /loss/FRt1/detection_loss‚2.0228209357995253 .loss/FBt1/detection_loss‚2.880071073770523N1µÎ_ Ù 0loss/FRt4/assignment_loss‚1.6844674562508206 2loss/SRqqt3/assignment_loss‚1.2396324376630432 1loss/FBt2/assignment_loss‚0.41430356763435855 1loss/SRqqt3/detection_loss‚1.7099792413851793 # _timestamp‚1.78840009465004e+09 /loss/FRt3/assignment_loss‚3.164890603963719 0loss/SRqqt4/detection_loss‚1.222224473146101 /loss/FBt4/detection_loss‚0.6505940780043602 epoch‚4 2loss/SRqqt2/assignment_loss‚1.6369057452237163 1loss/FBt3/assignment_loss‚0.27181429635735793 .loss/FBt3/detection_loss‚3.495490332444509 2loss/SRqqt1/assignment_loss‚1.3754775579353529 /loss/FRt3/detection_loss‚2.5320963707081106 1loss/SRqqt1/detection_loss‚1.8214756729347366 &loss/total_loss‚45.448155417716926 _step‚487 -loss/FRt4/detection_loss‚1.77002843764593 1loss/SRqqt2/detection_loss‚2.6019603510697684 2loss/SRqqt4/assignment_loss‚0.8423101702705026 1loss/FBt4/assignment_loss‚0.15100753534511507 /loss/FRt2/detection_loss‚3.3791981261709463 trainer/global_step‚5949 _runtime‚4719.286760612 0loss/FRt1/assignment_loss‚3.0274462894751477 /loss/FRt2/assignment_loss‚4.442247141962466 /loss/FRt1/detection_loss‚2.0228209357995253 .loss/FBt1/detection_loss‚2.880071073770523 1loss/FBt1/assignment_loss‚0.48810924325759214 .loss/FBt2/detection_loss‚2.623604279298049烿»˜’¡‚(:‡ ä£ãÔ‡Û©® gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚119.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9466 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02Ë{¢‚(:ð ä£ãÔ˪¥±* disk./.usagePercent‚15.223368980095476 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda2.out‚0) proc.memory.percent‚16.44001007080078 disk.dm-0.in‚ 370.58984375# disk.dm-0.out‚25224.25048828125 disk.nvme0n1.in‚ 370.58984375 disk.dm-1.in‚ 0.0546875 proc.memory.rssMB‚ 2693.53125 proc.cpu.threads‚17 network.sent‚ 1.4041488e+07 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda3.out‚ 7601.9453125" disk.nvme0c0n1.in‚ 370.58984375( disk.nvme0c0n1.out‚25224.25048828125 disk.drbd2266.in‚0 disk.rbd1.in‚0 cpu‚25.325783804515634$ disk./.usageGB‚226.8391990661621 disk.sda1.in‚0 disk.sda1.out‚0 disk.dm-1.out‚ 7601.9453125 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 14.3779296875 disk.sda.in‚ 0.0546875 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.sda3.in‚ 0.0546875& disk.nvme0n1.out‚25224.25048828125 disk.sda.out‚ 7601.9453125# memory_percent‚93.7582015991211& proc.memory.availableMB‚ 1022.65625 network.recv‚ 1.714794e+06 disk.drbd1716.in‚0Bd˜6£‚(:’ ó£ãÔæü ® gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚129.709' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.941799999999997 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]H¢Â€¤‚(:õ ó£ãÔ³²ö° disk.sda.in‚ 0.0546875 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 7602.33203125$ memory_percent‚93.75829696655273 network.sent‚ 1.4150191e+07 disk.drbd1716.out‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.0546875 disk.dm-1.out‚ 7602.33203125 disk.drbd2080.out‚0 disk.sda3.in‚ 0.0546875( disk.nvme0c0n1.out‚25224.63525390625 cpu‚25.324919557198214 proc.cpu.threads‚17 network.recv‚ 1.721575e+06 disk.sda.out‚ 7602.33203125 disk.drbd2265.out‚0& disk.nvme0n1.out‚25224.63525390625 disk.drbd2266.out‚0# disk.dm-0.out‚25224.63525390625 disk.rbd1.out‚ 14.3779296875' proc.memory.availableMB‚ 1022.640625 proc.memory.rssMB‚ 2693.53125) proc.memory.percent‚16.44001007080078* disk./.usagePercent‚15.223372820208489$ disk./.usageGB‚226.8392562866211" disk.nvme0c0n1.in‚ 370.58984375 disk.nvme0n1.in‚ 370.58984375 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 370.58984375 disk.rbd1.in‚0 disk.sda2.in‚0 disk.drbd2080.in‚0É~Y2¥- _wandbruntime‚4754  _runtime‚4754xæ@ɦà ' lr-AdamW/pg2‚0.00012756359224008266 trainer/global_step‚5999 % _timestamp‚1.7884001297712505e+09 _runtime‚4754.408083188 _step‚488 ' lr-AdamW/pg1‚0.00012756359224008266®ÕÈÎ§È ' lr-AdamW/pg1‚0.00012756359224008266 ' lr-AdamW/pg2‚0.00012756359224008266 trainer/global_step‚5999 % _timestamp‚1.7884001297712505e+09 _runtime‚4754.408083188 _step‚488èKŠ2¨- _wandbruntime‚4754  _runtime‚4754/[«p©ØO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚168382878442496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176830734589952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 eQDeviceStatsMonitor.on_train_batch_sta9:IÕn%rt/inactive_split_bytes.large_pool.allocated‚178429000906752 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚168603876560404 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚160180162503680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8423714056724 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79834735 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚187023522722304 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 trainer/global_step‚5999 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚176830921592832 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚176830921592832 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8448043150336 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚489 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120989049 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33926529 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 _runtime‚4754.408957818 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚168382878442496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚168603694399640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121009087 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 $ _timestamp‚1.788400129771965e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199671278 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚187023279809024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8447874150400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45908206 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚160180145464320 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78662191ïNçƒã'ªÝO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚178428976966656 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚78662189 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8447874150400 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚78662189 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8447874150400 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚45908206 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚160180145464320 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚78662191 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚168382878442496 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚176830734589952 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚178429000906752 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚168603876560404 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚160180162503680 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8423714056724 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚79834735 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚187023522722304 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 trainer/global_step‚5999 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚176830921592832 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚176830921592832 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8448043150336 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚489 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 120989049 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚33926529 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 _runtime‚4754.408957818 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚168382878442496 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚168603694399640 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 121009087 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 $ _timestamp‚1.788400129771965e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 199671278 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚187023279809024 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168631026746520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199703750 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78674848 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚176859584109056 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚178457889914880 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚176859584109056 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78674848 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79846894 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚160206088646656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚176859397092864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8595718765568 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45915583 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8424938099864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚553 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚490 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚168410147807232 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 _runtime‚4755.087202354 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 199683684 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚168410147807232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚187053608680448 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121008836 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚187053849483264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45915581 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8449436301824 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚168410129804288 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121028900 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8449436301824 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79847449 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096?LìZ¯èM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚176859397092864 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8595718765568 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚45915583 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8424938099864 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚553 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 _step‚490 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚168410147807232 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_by…OC—”!tes.small_pool.current‚ 216862720 _runtime‚4755.087202354 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 199683684 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚168410147807232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚187053608680448 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 121008836 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚187053849483264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚45915581 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8449436301824 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚168410129804288 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 121028900 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8449436301824 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚79847449 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 121008836 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8449267288576 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 121028900 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚168631026746520 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 199703750 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚78674848 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚176859584109056 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚178457889914880 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚176859584109056 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚78674848 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚79846894 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚160206088646656 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0êø0-T2°- _wandbruntime‚4755  _runtime‚4755…Ã*U±Ï /loss/FRt2/assignment_loss‚4.158784415410913 /loss/FBt1/detection_loss‚2.2564321495592594 .loss/FRt1/detection_loss‚1.971926698559209 /loss/FBt4/detection_loss‚0.6402276031501395 /loss/FRt2/detection_loss‚3.1636976729268613 1loss/SRqqt3/detection_loss‚1.9700989979644155 /loss/FBt3/detection_loss‚2.6308996847697665 % _timestamp‚1.7884001304515536e+09 .loss/FRt1/assignment_loss‚3.31929613100855 0loss/FRt4/assignment_loss‚1.7409503949540002 2loss/SRqqt3/assignment_loss‚1.4625181359737072 0loss/SRqqt1/detection_loss‚1.956488680332265 .loss/FBt2/detection_loss‚3.425321380297343 epoch‚4 2loss/SRqqt1/assignment_loss‚1.3685764026768663 0loss/FBt3/assignment_loss‚0.3647026254614372 1loss/FBt4/assignment_loss‚0.18012053190406468 .loss/FRt3/detection_loss‚2.868677569553256 1loss/SRqqt2/detection_loss‚2.5414239137337127 1loss/SRqqt4/detection_loss‚0.9372156729845151 2loss/SRqqt2/assignment_loss‚1.9525830283247192 0loss/FBt2/assignment_loss‚0.3096813224249975 /loss/FRt4/detection_loss‚1.7133648239874415 trainer/global_step‚5999 _runtime‚ 4755.08862616 _step‚491 %loss/total_loss‚45.88414498789574 0loss/FRt3/assignment_loss‚2.9688969049602747 /loss/SRqqt4/assignment_loss‚1.1289026823201 0loss/FBt1/assignment_loss‚0.8533575646579266F!rNZ²Ô % _timestamp‚1.7884001304515536e+09 .loss/FRt1/assignment_loss‚3.31929613100855 0loss/FRt4/assignment_loss‚1.7409503949540002 2loss/SRqqt3/assignment_loss‚1.4625181359737072 0loss/SRqqt1/detection_loss‚1.956488680332265 .loss/FBt2/detection_loss‚3.425321380297343 epoch‚4 2loss/SRqqt1/assignment_loss‚1.3685764026768663 0loss/FBt3/assignment_loss‚0.3647026254614372 1loss/FBt4/assignment_loss‚0.18012053190406468 .loss/FRt3/detection_loss‚2.868677569553256 1loss/SRqqt2/detection_loss‚2.5414239137337127 1loss/SRqqt4/detection_loss‚0.9372156729845151 2loss/SRqqt2/assignment_loss‚1.9525830283247192 0loss/FBt2/assignment_loss‚0.3096813224249975 /loss/FRt4/detection_loss‚1.7133648239874415 trainer/global_step‚5999 _runtime‚ 4755.08862616 _step‚491 %loss/total_loss‚45.88414498789574 0loss/FRt3/assignment_loss‚2.9688969049602747 /loss/SRqqt4/assignment_loss‚1.1289026823201 0loss/FBt1/assignment_loss‚0.8533575646579266 /loss/FRt2/assignment_loss‚4.158784415410913 /loss/FBt1/detection_loss‚2.2564321495592594 .loss/FRt1/detection_loss‚1.971926698559209 /loss/FBt4/detection_loss‚0.6402276031501395 /loss/FRt2/detection_loss‚3.1636976729268613 1loss/SRqqt3/detection_loss‚1.9700989979644155 /loss/FBt3/detection_loss‚2.6308996847697665ëï$ýí³‚(:… ‘¤ãÔô’Ö® gpu.0.gpu‚29.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚146.42' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚179090778162176 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 4541568000 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚46566046 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 123925592 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 122267395 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 202036786 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 4624220160 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚80981519 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚199 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚985 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚199 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 8676838998528 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 269877760 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚180719547297792 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 115961344 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚199 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚79769391 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 8529260149248 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚170561645321216 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 202052765 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚180719571237888 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚416 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 122267395 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚170561645321216 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 8529132840960 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2460 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 5041553408 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚170755258187960 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚217 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚79769393 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚34415471 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 202052765 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚179090778162176 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 4541568000 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚46566046 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 123925592 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 122267395 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 202036786 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 4624220160 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚80981519 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚199 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚985 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚199 dRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚ 8676838998528 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 269877760 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚180719547297792 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 115961344 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚199 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 QDDeviceStatsMonitor.on_validation_batch_start/active.lar ‹¾ïge_pool.freed‚79769391 `NDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚ 8529260149248 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚170561645321216 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 202052765 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚180719571237888 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚416 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 122267395 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚170561645321216 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.freed‚ 8529132840960 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2460 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 5041553408 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚170755258187960 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚217 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚79769393 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚34415471 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 202052765 FDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 8677188134400 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 _runtime‚4836.676746181 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 202070873 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 4912020992 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚6044 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚179110137795072 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚170773648312912 % _timestamp‚1.7884002120395408e+09 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚46572548 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚180739150863360 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 bPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚ 8677188134400 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 _runtime‚4836.676746181 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 202070873 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 4912020992 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚6044 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚179110137795072 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚170773648312912 % _timestamp‚1.7884002120395408e+09 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚46572548 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚180739150863360 B&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0015873016091063619 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9801009893417358 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.28409090638160706 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 % _timestamp‚1.7884002245194902e+09 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9584795236587524 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5416666865348816 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.49903056025505066 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5100905895233154 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1506488025188446 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1197371706366539 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.42317602038383484 particle/accuracy_5_of_5‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21591484546661377 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.020698923617601395 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1502561867237091 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5070323348045349 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3611111044883728 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1527777761220932 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6469519138336182 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4166666865348816 jet/accuracy_4_of_6‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0130328219383955 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 particle/accuracy_2_of_2‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2708333432674408 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20112736523151398 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3338383734226227 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 +jet/accuracy_4_of_4‚0.00511051993817091 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6013888716697693 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5196266174316406 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5007495284080505 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3738095164299011 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4662356972694397 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.38856586813926697 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.48148149251937866 +jet/accuracy_2_of_4‚0.34785696864128113 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 jet/accuracy_3_of_7‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9903886914253235 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.24040402472019196 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt1SRqqt2FBt/event_purity‚0.0258620698004961 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20438937842845917 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2736992835998535 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3464559316635132 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.692289412021637 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1859978288412094 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.47077780961990356 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9533333778381348 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998068630695343 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5563172698020935 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08761268109083176 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1309524029493332 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_1_of_1‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.11555555462837219 jet/accuracy_6_of_6‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.43179067969322205 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3437812924385071 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5208333134651184 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12228327244520187 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.37710660696029663 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0847603902220726 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13417206704616547 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_1_of_3‚0.8477650880813599 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4729333817958832 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4574030637741089 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5146785974502563 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14313216507434845 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9600694179534912 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9735023379325867 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ;"Purity/3FRt0SRqqt*FBt/event_purity‚0.011111111380159855 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.023795630782842636 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11205660551786423 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ."Purity/3FRt0SRqqt1FBt/event_purity‚0.03125 +jet/accuracy_3_of_6‚0.11538461595773697 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5416666865348816 5 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.98255455493927 particle/accuracy_2_of_3‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9773584604263306 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3374268710613251 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4732699394226074 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.29005753993988037 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5338366627693176 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.972289502620697 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.39400702714920044 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.42990219593048096 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2549549639225006 particle/accuracy_3_of_3‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9733560085296631 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.23850573599338531 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5019841194152832 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2748015820980072 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26849523186683655 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1457197517156601 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3915170729160309 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2708333432674408 particle/accuracy_4_of_6‚1 particle/accuracy_5_of_7‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8446438312530518 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.42797860503196716 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.28012973070144653 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1527777761220932 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964266419410706 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.15857599675655365 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4840124249458313 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1534840613603592 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.029069768264889717 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11084704846143723 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31385067105293274 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.45970961451530457 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5427190661430359 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964838027954102 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24167180061340332 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3081445097923279 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 +particle/specificity‚0.9473994374275208 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ;"Purity/3FRt*SRqqt1FBt/event_purity‚0.008333333767950535 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.021275494247674942 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0023419205099344254 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9920859336853027 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.35553640127182007 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3495495319366455 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 jet/accuracy_2_of_7‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.437492698431015 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.49163904786109924 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5061655640602112 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_2_of_7‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.503624677658081 epoch‚4 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.38235294818878174 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.025157233700156212 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_7_of_7‚1 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 6 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.334077388048172 +jet/accuracy_2_of_5‚0.40086403489112854 7validation_average_jet_accuracy‚0.42551693320274353 *jet/accuracy_1_of_6‚0.9807692170143127 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9906415939331055 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14152786135673523 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7121211886405945 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 particle/accuracy_6_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4044725298881531 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4533333480358124 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9734195470809937 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3027777671813965 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_4‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2768518626689911 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.2103174775838852 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.35798946022987366 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5897435545921326 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *jet/accuracy_1_of_2‚0.7344609498977661 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3580247461795807 *jet/accuracy_2_of_6‚0.4166666865348816 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20300379395484924 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6228070259094238 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 +jet/accuracy_3_of_5‚0.09150025993585587 jet/accuracy_6_of_7‚0 particle/accuracy_1_of_7‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1394951343536377 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1591954231262207 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.34210526943206787 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 _step‚494 particle/accuracy_3_of_4‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13840579986572266 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 particle/accuracy_4_of_7‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4451754689216614 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10777586698532104 ,jet/accuracy_3_of_3‚0.022413836792111397 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5283898115158081 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5170260667800903 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13229656219482422 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14176245033740997 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.49163904786109924 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14313216507434845 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9941393733024597 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9895169138908386 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23452991247177124 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 particle/accuracy_3_of_5‚1 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.3125 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13094088435173035 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36088666319847107 particle/accuracy_2_of_4‚1 'particle/f_score‚0.3675687909126282 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3786141276359558 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9875941872596741 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20597688853740692 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_4‚0.8825921416282654 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4950534999370575 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5554827451705933 particle/accuracy_2_of_5‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5618664026260376 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9858510494232178 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006533942651003599 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.47477954626083374 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9972696304321289 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3437499701976776 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06568917632102966 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4987874925136566 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.34744077920913696 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.032004281878471375 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.514162540435791 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 )jet/accuracy_3_of_4‚0.066157266497612 *jet/accuracy_1_of_5‚0.9211596846580505 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05758756771683693 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5196428894996643 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02777777798473835 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.031760524958372116 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8253811001777649 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_1_of_6‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1741899698972702 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9889446496963501 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 +particle/sensitivity‚0.2894013822078705 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 particle/accuracy_6_of_7‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9827795624732971 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16200271248817444 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15530787408351898 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4302992820739746 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4568618834018707 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.999007÷ Ütð 9402923584 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5474693179130554 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.019416799768805504 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04122408106923103 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5022244453430176 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_7‚0.5 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.3125 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5182039737701416 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1417863517999649 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23941156268119812 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4885057508945465 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.41689789295196533 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15496613085269928 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.13991768658161163 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 )loss/val_total_loss‚45.92700579668631 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.39755240082740784 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.45403024554252625 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18513134121894836 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1473684161901474 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18599765002727509 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24410776793956757 *jet/accuracy_2_of_3‚0.2373892068862915 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993895292282104 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3795138895511627 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 jet/accuracy_5_of_6‚0 jet/accuracy_5_of_7‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4723183810710907 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.25826048851013184 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.20454543828964233 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.45721179246902466 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07113095372915268Þv•sÇŸ§ 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4723183810710907 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.25826048851013184 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.20454543828964233 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.45721179246902466 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07113095372915268 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9196218252182007 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3679012358188629 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 +jet/accuracy_1_of_1‚0.47409355640411377 particle/accuracy_4_of_5‚1 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2916666865348816 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.987857460975647 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.3611111044883728 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.234375 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 particle/accuracy_3_of_7‚1 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 :"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.47658923268318176 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 trainer/global_step‚6044 _runtime‚4849.158318441 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.40625 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5258610844612122 6 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.987742006778717 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.013401469215750694 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.30000001192092896 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.3661220073699951 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9877922534942627 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17709028720855713 ,jet/accuracy_4_of_5‚0.005376344081014395 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4266666769981384 <"Purity/2FRt1SRqqt0FBt/event_purity‚0.0031746032182127237 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41349056363105774 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.011772487312555313 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18116052448749542 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.17105263471603394 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1913580447435379 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4311131536960602 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9767857193946838 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1853688508272171 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5120024085044861 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.05080030858516693 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14318330585956573 9"Purity/*FRt*SRqqt3FBt/event_purity‚0.1666666716337204 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013618317432701588 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0015873016091063619 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9801009893417358 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.28409090638160706 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 % _timestamp‚1.7884002245194902e+09 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9584795236587524 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5416666865348816 :"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.49903056025505066 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5100905895233154 7 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.1506488025188446 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1197371706366539 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.42317602038383484 particle/accuracy_5_of_5‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.21591484546661377 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.020698923617601395 7 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.1502561867237091 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5070323348045349 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.3611111044883728 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1527777761220932 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6469519138336182 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4166666865348816 jet/accuracy_4_of_6‚0 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 9"Purity/2FRt0SRqqt0FBt/event_purity‚0.0130328219383955 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 particle/accuracy_2_of_2‚1 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2708333432674408 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20112736523151398 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.3338383734226227 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 +jet/accuracy_4_of_4‚0.00511051993817091 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_7‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6013888716697693 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5196266174316406 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5007495284080505 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 7 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.3738095164299011 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4662356972694397 8 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.38856586813926697 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.48148149251937866 +jet/accuracy_2_of_4‚0.34785696864128113 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 jet/accuracy_3_of_7‚0 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9903886914253235 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.24040402472019196 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt1SRqqt2FBt/event_purity‚0.0258620698004961 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20438937842845917 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2736992835998535 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3464559316635132 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8"Purity/*FRt0SRqqt1FBt/event_purity‚0.692289412021637 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1859978288412094 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.47077780961990356 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.8947368264198303 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9533333778381348 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.2857142984867096 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998068630695343 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5563172698020935 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.08761268109083176 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1309524029493332 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 particle/accuracy_1_of_1‚1 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.11555555462837219 jet/accuracy_6_of_6‚0 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.43179067969322205 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3437812924385071 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5208333134651184 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12228327244520187 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.37710660696029663 9"Purity/1FRt*SRqqt2FBt/event_purity‚0.0847603902220726 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13417206704616547 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_1_of_3‚0.8477650880813599 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.30000001192092896 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4729333817958832 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4574030637741089 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5146785974502563 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.14313216507434845 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9600694179534912 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9735023379325867 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ;"Purity/3FRt0SRqqt*FBt/event_purity‚0.011111111380159855 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.023795630782842636 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11205660551786423 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ."Purity/3FRt0SRqqt1FBt/event_purity‚0.03125 +jet/accuracy_3_of_6‚0.11538461595773697 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5416666865348816 5 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.98255455493927 particle/accuracy_2_of_3‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9773584604263306 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3374268710613251 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 ( Purity/1FRt*SRqqt4FBt/FRt_purity‚0.5 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.4732699394226074 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.29005753993988037 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.4285714328289032 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5338366627693176 6 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.972289502620697 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.39400702714920044 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.42990219593048096 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2549549639225006 particle/accuracy_3_of_3‚1 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9733560085296631 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.23850573599338531 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5019841194152832 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2748015820980072 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26849523186683655 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1457197517156601 & Purity/*FRt1SRqqt4FBt/FRt_purity‚1 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.3915170729160309 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2708333432674408 particle/accuracy_4_of_6‚1 particle/accuracy_5_of_7‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 (particle/accuracy‚0.8446438312530518 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.42797860503196716 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.28012973070144653 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚1 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1527777761220932 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964266419410706 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.15857599675655365 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4840124249458313 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1534840613603592 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.029069768264889717 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_2‚0.11084704846143723 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31385067105293274 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 :"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.45970961451530457 & Purity/*FRt3SRqqt2FBt/FRt_purity‚1 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9107142686843872 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5427190661430359 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9964838027954102 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.24167180061340332 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3081445097923279 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 +particle/specificity‚0.9473994374275208 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 ;"Purity/3FRt*SRqqt1FBt/event_purity‚0.008333333767950535 & Purity/3FRt4SRqqt*FBt/FRt_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.021275494247674942 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0023419205099344254 +"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.25 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9920859336853027 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.35553640127182007 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3495495319366455 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 jet/accuracy_2_of_7‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.0625 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.437492698431015 :"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.49163904786109924 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5061655640602112 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_2_of_7‚1 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 8"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.503624677658081 epoch‚4 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.38235294818878174 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.025157233700156212 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_7_of_7‚1 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 6 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.334077388048172 +jet/accuracy_2_of_5‚0.40086403489112854 7validation_average_jet_accuracy‚0.42551693320274353 *jet/accuracy_1_of_6‚0.9807692170143127 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.3333333432674408 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9906415939331055 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.14152786135673523 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7121211886405945 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 particle/accuracy_6_of_6‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4044725298881531 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4533333480358124 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9734195470809937 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3027777671813965 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_4‚1 7 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.2768518626689911 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.2103174775838852 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.35798946022987366 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5897435545921326 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 *jet/accuracy_1_of_2‚0.7344609498977661 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3580247461795807 *jet/accuracy_2_of_6‚0.4166666865348816 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20300379395484924 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6228070259094238 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4000000059604645 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚1 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 +jet/accuracy_3_of_5‚0.09150025993585587 jet/accuracy_6_of_7‚0 particle/accuracy_1_of_7‚1 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1394951343536377 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1591954231262207 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.34210526943206787 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 _step‚494 particle/accuracy_3_of_4‚1 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13840579986572266 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_6‚1 particle/accuracy_4_of_7‚1 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4451754689216614 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10777586698532104 ,jet/accuracy_3_of_3‚0.022413836792111397 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5283898115158081 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5170260667800903 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13229656219482422 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.14176245033740997 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt0FBt/event_purity‚0.49163904786109924 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.14313216507434845 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9941393733024597 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9895169138908386 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23452991247177124 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.37037035822868347 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7424241900444031 particle/accuracy_3_of_5‚1 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.3125 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13094088435173035 =&Purity/1FRt1SRqqt1FBt/evM{‹Q¤ ent_proportion‚0.0392485111951828 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.36088666319847107 particle/accuracy_2_of_4‚1 'particle/f_score‚0.3675687909126282 9"Purity/*FRt0SRqqt*FBt/event_purity‚0.3786141276359558 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9365079998970032 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9875941872596741 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20597688853740692 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_1_of_4‚0.8825921416282654 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.4950534999370575 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5554827451705933 particle/accuracy_2_of_5‚1 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5618664026260376 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9858510494232178 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.006533942651003599 :"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.47477954626083374 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9972696304321289 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3437499701976776 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06568917632102966 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4987874925136566 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.34744077920913696 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.032004281878471375 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.514162540435791 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 )jet/accuracy_3_of_4‚0.066157266497612 *jet/accuracy_1_of_5‚0.9211596846580505 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.05758756771683693 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5196428894996643 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.0625 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02777777798473835 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.031760524958372116 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8253811001777649 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_1_of_6‚1 & Purity/3FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ( Purity/*FRt*SRqqt4FBt/FRt_purity‚0.5 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1741899698972702 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9889446496963501 9"Purity/0FRt0SRqqt1FBt/event_purity‚0.9990079402923584 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 +particle/sensitivity‚0.2894013822078705 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9270833730697632 particle/accuracy_6_of_7‚1 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9827795624732971 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16200271248817444 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15530787408351898 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4302992820739746 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4568618834018707 7 Purity/0FRt0SRqqt1FBt/FBt_purity‚0.9990079402923584 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5474693179130554 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.019416799768805504 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04122408106923103 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5022244453430176 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 jet/accuracy_1_of_7‚0.5 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.3125 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5182039737701416 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1417863517999649 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 +"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.23941156268119812 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.4885057508945465 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ) Purity/2FRt3SRqqt1FBt/FRt_purity‚0.25 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.41689789295196533 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15496613085269928 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.13991768658161163 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 )loss/val_total_loss‚45.92700579668631 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.39755240082740784 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.45403024554252625 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.18513134121894836 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1473684161901474 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.18599765002727509 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24410776793956757 *jet/accuracy_2_of_3‚0.2373892068862915 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FBt_purity‚0.9993895292282104 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3795138895511627 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 jet/accuracy_5_of_6‚0 jet/accuracy_5_of_7‚0 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0M2È- _wandbruntime‚4854  _runtime‚4854Ô£ÚÅÇÉÁ & lr-AdamW/pg1‚0.0001260370541831034 & lr-AdamW/pg2‚0.0001260370541831034 trainer/global_step‚6049 % _timestamp‚1.7884002301977236e+09 _runtime‚4854.834563253 _step‚495W¹úžÌÊÆ _step‚495 & lr-AdamW/pg1‚0.0001260370541831034 & lr-AdamW/pg2‚0.0001260370541831034 trainer/global_step‚6049 % _timestamp‚1.7884002301977236e+09 _runtime‚4854.834563253ïÎg(2Ë- _wandbruntime‚4854  _runtime‚4854Û’Þ'ÌØO _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8687379226624 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚79969997 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179474393404928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚4854.835446279 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚470 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34503578 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚181109649008640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.788400230198396e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81190604 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202419631 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8539727023616 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122469674 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122449636 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46686554 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34504048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚181109672948736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218973184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚171120039522226 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚171120221682990 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚162605102461952 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8539558023680 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8539558023680 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚179474393404928 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚179474580407808 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46686556 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189797028235264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8515102181678 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122469674 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202439671 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81190132 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170934853384192 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189797271148544 trainer/global_step‚6049 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _step‚496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚179474580407808 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122449636 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79969997 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170934853384192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79969995 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8514937060274 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170934835381248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8539727023616 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚179474580407808 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚46686556 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚189797028235264 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8515102181678 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 122469674 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 202439671 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81190132 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚170934853384192 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚189797271148544 trainer/global_step‚6049 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _step‚496 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚179474580407808 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 122449636 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚79969997 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚170934853384192 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚79969995 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8514937060274 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚170934835381248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8539727023616 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚179474393404928 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚4854.835446279 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚470 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34503578 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚181109649008640 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 $ _timestamp‚1.788400230198396e+09 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81190604 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 202419631 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8539727023616 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 122469674 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 122449636 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚46686554 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34504048 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚181109672948736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218973184 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚171120039522226 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚472 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚171120221682990 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚162605102461952 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8539558023680 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217ð‚õãÊ2Î- _wandbruntime‚4855  _runtime‚4855«.Îj3 ÏãM H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46693931 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34509360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8540951161856 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170962104745984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81202798 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79982654 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122469423 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚171147371869106 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122489487 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8689011371520 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46693933 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8688788217344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚179503242924032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81203293 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚495 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚181138585896960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8541120175104 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚493 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚171147554030078 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223154176 XIDeviceStatsMonitor.on_train_batch_bC{Ù¶end/reserved_bytes.large_pool.allocated‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202472143 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79982656 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170962122748928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚162631062683648 trainer/global_step‚6049 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170962104745984 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202472143 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚181138561956864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 % _timestamp‚1.7884002308797023e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8540951161856 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122469423 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚179503055907840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202452077 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚179503242924032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34508867 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8516491346430 _step‚497 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚493 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚171147554030078 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 223154176 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 202472143 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚79982656 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚170962122748928 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚162631062683648 trainer/global_step‚6049 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚170962104745984 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 202472143 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚181138561956864 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 % _timestamp‚1.7884002308797023e+09 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8540951161856 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 122469423 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚179503055907840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 202452077 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚179503242924032 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34508867 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8516491346430 _step‚497 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚46693931 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34509360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8540951161856 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚170962104745984 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81202798 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚79982654 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 122469423 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚171147371869106 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 122489487 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8689011371520 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚46693933 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8688788217344 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚179503242924032 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81203293ñ…Ñé2Ñ- _wandbruntime‚4855  _runtime‚4855þÓ‚VÒÐ 0loss/FBt2/assignment_loss‚0.5238704660709852 /loss/FRt4/detection_loss‚1.7095878068357706 0loss/SRqqt2/detection_loss‚2.461424246430397 .loss/FBt3/detection_loss‚3.294179581105709 trainer/global_step‚6049 % _timestamp‚1.7884002308813527e+09 _runtime‚4855.518100532 0loss/FRt2/assignment_loss‚3.6412342063018253 -loss/FRt4/assignment_loss‚1.4926661302646 1loss/SRqqt1/assignment_loss‚1.764030179917449 2loss/SRqqt4/assignment_loss‚0.8612890917334765 1loss/FBt4/assignment_loss‚0.16972090432836007 .loss/FRt3/detection_loss‚2.527185360459905 /loss/FRt3/assignment_loss‚2.368535451198879 1loss/SRqqt4/detection_loss‚1.0913072712128125 /loss/FBt4/detection_loss‚0.7541538744245644 %loss/total_loss‚44.34867943304198 /loss/FRt1/detection_loss‚2.2048169192002742 /loss/FRt2/detection_loss‚2.9393112478511676 0loss/FRt1/assignment_loss‚3.3462859769256745 0loss/FBt3/assignment_loss‚0.7491577193140984 1loss/SRqqt1/detection_loss‚2.0256201348653655 /loss/FBt2/detection_loss‚2.5320454285695004 epoch‚5 _step‚498 2loss/SRqqt2/assignment_loss‚1.8554740813043382 0loss/FBt1/assignment_loss‚0.3822273263474926 0loss/SRqqt3/detection_loss‚1.865577154689365 .loss/FBt1/detection_loss‚2.327943342924118 2loss/SRqqt3/assignment_loss‚1.4610355307658514ɼM[ÓÕ /loss/FRt1/detection_loss‚2.2048169192002742 /loss/FRt2/detection_loss‚2.9393112478511676 0loss/FRt1/assignment_loss‚3.3462859769256745 0loss/FBt3/assignment_loss‚0.7491577193140984 1loss/SRqqt1/detection_loss‚2.0256201348653655 /loss/FBt2/detection_loss‚2.5320454285695004 epoch‚5 _step‚498 2loss/SRqqt2/assignment_loss‚1.8554740813043382 0loss/FBt1/assignment_loss‚0.3822273263474926 0loss/SRqqt3/detection_loss‚1.865577154689365 .loss/FBt1/detection_loss‚2.327943342924118 2loss/SRqqt3/assignment_loss‚1.4610355307658514 0loss/FBt2/assignment_loss‚0.5238704660709852 /loss/FRt4/detection_loss‚1.7095878068357706 0loss/SRqqt2/detection_loss‚2.461424246430397 .loss/FBt3/detection_loss‚3.294179581105709 trainer/global_step‚6049 % _timestamp‚1.7884002308813527e+09 _runtime‚4855.518100532 0loss/FRt2/assignment_loss‚3.6412342063018253 -loss/FRt4/assignment_loss‚1.4926661302646 1loss/SRqqt1/assignment_loss‚1.764030179917449 2loss/SRqqt4/assignment_loss‚0.8612890917334765 1loss/FBt4/assignment_loss‚0.16972090432836007 .loss/FRt3/detection_loss‚2.527185360459905 /loss/FRt3/assignment_loss‚2.368535451198879 1loss/SRqqt4/detection_loss‚1.0913072712128125 /loss/FBt4/detection_loss‚0.7541538744245644 %loss/total_loss‚44.34867943304198òá“’Ô‚(:‡ ë¤ãԥĮ gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚149.372' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.8744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'´‡Õ‚(:ü ë¤ãÔ½¡´± disk.rbd0.in‚0 disk.rbd1.out‚ 14.3779296875 disk.dm-1.in‚ 0.0546875 disk.drbd2265.out‚0 cpu‚25.381928102854374 network.recv‚ 1.760308e+06% disk./.usageGB‚226.84028244018555 disk.sda3.in‚ 0.0546875 disk.rbd1.in‚0 disk.sda2.out‚0* proc.memory.percent‚16.442084312438965* disk./.usagePercent‚15.223441686235201 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.drbd2265.in‚0# proc.memory.rssMB‚ 2693.87109375 disk.drbd2080.out‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 7610.34765625 disk.dm-0.in‚ 370.58984375 disk.nvme0n1.in‚ 370.58984375 disk.sda.in‚ 0.0546875 disk.sda.out‚ 7610.34765625) proc.memory.availableMB‚ 1022.10546875 disk.sda1.in‚0( disk.nvme0c0n1.out‚25359.73486328125$ memory_percent‚93.76156330108643 network.sent‚ 1.4482925e+07 disk.sda1.out‚0 proc.cpu.threads‚17# disk.dm-0.out‚25359.73486328125 disk.sda3.out‚ 7610.34765625" disk.nvme0c0n1.in‚ 370.58984375& disk.nvme0n1.out‚25359.73486328125 disk.drbd1716.out‚0 disk.drbd2266.out‚0]m»Ö‚(:’ ú¤ãÔ裸® gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.661' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.132199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0í“j…ׂ(:ú ú¤ãÔð‘ð± network.recv‚ 1.765935e+06 disk.sda1.out‚0 disk.sda3.in‚ 0.0546875" disk.nvme0c0n1.in‚ 370.58984375 disk.drbd2265.in‚0 disk.drbd2080.in‚0 proc.cpu.threads‚17 network.sent‚ 1.462762e+07 disk.sda3.out‚ 7611.07421875* disk./.usagePercent‚15.223405845180407% disk./.usageGB‚226.83974838256836 disk.sda1.in‚0 disk.drbd1716.out‚0$ memory_percent‚93.75777244567871 cpu‚25.380907049405238 disk.rbd0.out‚0# disk.dm-0.out‚25372.03173828125 disk.sda.out‚ 7611.07421875 disk.drbd1716.in‚0 disk.dm-1.out‚ 7611.07421875 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 14.3779296875 disk.dm-0.in‚ 370.58984375( disk.nvme0c0n1.out‚25372.03173828125 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2693.87109375& disk.nvme0n1.out‚25372.03173828125( proc.memory.availableMB‚ 1022.7265625* proc.memory.percent‚16.442084312438965 disk.rbd0.in‚0 disk.nvme0n1.in‚ 370.58984375 disk.sda2.out‚0 disk.dm-1.in‚ 0.0546875 disk.drbd2266.in‚0 disk.sda.in‚ 0.0546875Ô@d’Ø‚(:‡ ‰¥ãÔ‡›Œ® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚149.372' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.8744 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08îɘ‚Ù‚(:÷ ‰¥ãÔß î°% disk./.usageGB‚226.83972930908203 disk.dm-0.in‚ 370.58984375" disk.dm-0.out‚25483.8037109375$ memory_percent‚93.20158958435059( proc.memory.availableMB‚ 1113.8515625 disk.dm-1.out‚ 7611.44921875 disk.sda1.in‚0 disk.sda3.in‚ 0.0546875 disk.sda3.out‚ 7611.44921875' disk.nvme0c0n1.out‚25483.8037109375 disk.rbd1.out‚ 14.3779296875* disk./.usagePercent‚15.223404565142735 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.sda.in‚ 0.0546875 network.sent‚ 1.4633661e+07 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.0546875 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 370.58984375 disk.sda.out‚ 7611.44921875* proc.memory.percent‚16.442084312438965 cpu‚25.37990798659032 network.recv‚ 1.770551e+06 disk.drbd2080.in‚0# proc.memory.rssMB‚ 2693.87109375 disk.nvme0n1.in‚ 370.58984375% disk.nvme0n1.out‚25483.8037109375 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda2.in‚0!ùñÂ2Ú- _wandbruntime‚4890  _runtime‚4890 ±'ÇÛÁ _step‚499 & lr-AdamW/pg1‚0.0001245066516369677 & lr-AdamW/pg2‚0.0001245066516369677 trainer/global_step‚6099 % _timestamp‚1.7884002659041727e+09 _runtime‚4890.541049378—ùîXÌÜÆ _step‚499 & lr-AdamW/pg1‚0.0001245066516369677 & lr-AdamW/pg2‚0.0001245066516369677 trainer/global_step‚6099 % _timestamp‚1.7884002659041727e+09 _runtime‚4890.541049378óÚ8›Ø2Ý- _wandbruntime‚4890  _runtime‚4890=úW7ÃÞÙO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123462376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81826970 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8609398346240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚172486852618198 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80602945 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47055404 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191313752298496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚180907532964352 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚180907532964352 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚172298303618048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _step‚500 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚524 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8609229346304 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34771040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚163902261578752 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚172298303618048 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123442338 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8758431938560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8584408878682 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80602947 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8609229346304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182555296419840 % _timestamp‚1.7884002659048984e+09 _runtime‚4890.541923397 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚180907719967232 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚180907719967232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81826444 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47055406 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚172298321620992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34771564 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚163902278618112 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0cÈ+ä'ßÞO PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚80602947 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8609229346304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚182555296419840 % _timestamp‚1.7884002659048984e+09 _runtime‚4890.541923397 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚180907719967232 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚180907719967232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚81826444 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47055406 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚172298321620992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚34771564 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚163902278618112 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 123462376 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚81826970 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8609398346240 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚172486852618198 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚80602945 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47055404 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚191313752298496 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚180907532964352 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚180907532964352 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚172298303618048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 _step‚500 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚524 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8609229346304 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚34771040 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚163902261578752 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚172298303618048 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 123442338 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8758431938560 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8584408878682ôN¥Æ2à- _wandbruntime‚4891  _runtime‚4891’Ab“é&áãM YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8610791497728 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8585963164838 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚172325590985728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204077729 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228219904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123462125 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚172325572982784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81838997 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8759642386944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204097795 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚163928204761088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204077729 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8610622484480 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚172514002804314 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚180936195467264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34776216 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚172325590985728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123482189 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚172514184965286 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7884002665802524e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47062781 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8610622484480 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81839511 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚191343851755008 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80615604 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34776728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚191344079974912 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8610791497728 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248õ¥ÿî&âèM ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚172325590985728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 204077729 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228219904 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 123462125 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚172325572982784 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚81838997 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8759642386944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 204097795 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚163928204761088 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 204077729 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8610622484480 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚172514002804314 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚180936195467264 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚34776216 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚172325590985728 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 123482189 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚172514184965286 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7884002665802524e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚47062781 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8610622484480 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚81839511 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚191343851755008 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚80615604 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚34776728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚191344079974912 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8610791497728 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8610791497728 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8585963164838õ:ÒŸŒ2ã- _wandbruntime‚4891  _runtime‚4891á?A+”äÔ /loss/FRt3/assignment_loss‚2.881719153980876 /loss/FBt2/assignment_loss‚0.645600978464032 /loss/FRt1/detection_loss‚1.9155112302909463 /loss/FBt2/detection_loss‚2.3927573760350547 _step‚502 2loss/SRqqt4/assignment_loss‚1.1353044853065952 0loss/FBt3/assignment_loss‚0.5306371169757437 /loss/FRt2/detection_loss‚3.2967171183338873 /loss/FRt2/assignment_loss‚4.42/ëåÆ5765752792358 1loss/FBt1/assignment_loss‚0.23337767861066822 0loss/SRqqt3/detection_loss‚1.501470162157427 /loss/FBt1/detection_loss‚3.0218962281942368 $loss/total_loss‚45.9924221398737 trainer/global_step‚6099 2loss/SRqqt2/assignment_loss‚2.2097218558192253 0loss/FRt1/assignment_loss‚3.1719108436067227 2loss/SRqqt1/assignment_loss‚1.6624304969673571 2loss/SRqqt3/assignment_loss‚1.6732852173675048 /loss/FRt3/detection_loss‚2.4684839200141817 0loss/SRqqt1/detection_loss‚1.843679607562397 _runtime‚4891.218711795 0loss/FRt4/assignment_loss‚2.0026160466904734 1loss/FBt4/assignment_loss‚0.17995246277992455 /loss/FRt4/detection_loss‚1.9119903772485023 0loss/SRqqt2/detection_loss‚2.792329141870141 /loss/FBt4/detection_loss‚0.5965135747307464 % _timestamp‚1.7884002665819736e+09 /loss/FBt3/detection_loss‚2.3771653337912126 1loss/SRqqt4/detection_loss‚1.1215859802834915 epoch‚5ÜŽ F_åÙ 1loss/SRqqt4/detection_loss‚1.1215859802834915 epoch‚5 /loss/FRt3/assignment_loss‚2.881719153980876 /loss/FBt2/assignment_loss‚0.645600978464032 /loss/FRt1/detection_loss‚1.9155112302909463 /loss/FBt2/detection_loss‚2.3927573760350547 _step‚502 2loss/SRqqt4/assignment_loss‚1.1353044853065952 0loss/FBt3/assignment_loss‚0.5306371169757437 /loss/FRt2/detection_loss‚3.2967171183338873 /loss/FRt2/assignment_loss‚4.425765752792358 1loss/FBt1/assignment_loss‚0.23337767861066822 0loss/SRqqt3/detection_loss‚1.501470162157427 /loss/FBt1/detection_loss‚3.0218962281942368 $loss/total_loss‚45.9924221398737 trainer/global_step‚6099 2loss/SRqqt2/assignment_loss‚2.2097218558192253 0loss/FRt1/assignment_loss‚3.1719108436067227 2loss/SRqqt1/assignment_loss‚1.6624304969673571 2loss/SRqqt3/assignment_loss‚1.6732852173675048 /loss/FRt3/detection_loss‚2.4684839200141817 0loss/SRqqt1/detection_loss‚1.843679607562397 _runtime‚4891.218711795 0loss/FRt4/assignment_loss‚2.0026160466904734 1loss/FBt4/assignment_loss‚0.17995246277992455 /loss/FRt4/detection_loss‚1.9119903772485023 0loss/SRqqt2/detection_loss‚2.792329141870141 /loss/FBt4/detection_loss‚0.5965135747307464 % _timestamp‚1.7884002665819736e+09 /loss/FBt3/detection_loss‚2.3771653337912126öûœ™æ‚(:’ ˜¥ãÔ–Âî gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚109.997' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚21.999399999999998 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0e„8ü†ç‚(:û ˜¥ãÔµ§± proc.cpu.threads‚17 disk.sda1.out‚0$ memory_percent‚93.74210834503174 network.recv‚ 1.774731e+06 disk.sda.in‚ 0.0546875 disk.dm-1.in‚ 0.0546875 disk.rbd1.in‚0& disk.nvme0n1.out‚25484.20068359375 cpu‚25.378901335514243* disk./.usagePercent‚15.223408917270817 disk.sda2.out‚0 network.sent‚ 1.463951e+07% disk./.usageGB‚226.83979415893555 disk.sda.out‚ 7611.98046875 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 7611.98046875 disk.drbd2266.out‚0# disk.dm-0.out‚25484.20068359375 disk.rbd0.out‚0 disk.sda3.in‚ 0.0546875 disk.rbd1.out‚ 14.3779296875) proc.memory.availableMB‚ 1025.29296875 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.dm-0.in‚ 370.58984375 disk.sda3.out‚ 7611.98046875( disk.nvme0c0n1.out‚25484.20068359375* proc.memory.percent‚16.442084312438965 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 370.58984375 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 370.58984375# proc.memory.rssMB‚ 2693.87109375a¶Ðrè‚(:… §¥ãÔÀ•® gpu.0.gpu‚34.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚123.51' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.702 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D~é‚(:ó §¥ãÔÝýí°* proc.memory.percent‚16.442084312438965* disk./.usagePercent‚15.223407893240681 disk.drbd1716.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.0546875 disk.rbd1.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.sda.in‚ 0.0546875 network.recv‚ 1.781287e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.0546875 disk.drbd2266.out‚0 disk.drbd2265.out‚0& disk.nvme0c0n1.out‚25485.583984375 cpu‚25.377906557442554 disk.dm-0.in‚ 370.58984375 disk.sda.out‚ 7612.47265625& proc.memory.availableMB‚ 934.109375 proc.cpu.threads‚17 disk.dm-1.out‚ 7612.47265625 disk.drbd2265.in‚0 disk.sda3.out‚ 7612.47265625 disk.nvme0n1.in‚ 370.58984375$ disk.nvme0n1.out‚25485.583984375$ memory_percent‚94.29864883422852% disk./.usageGB‚226.83977890014648 disk.drbd2266.in‚0! disk.dm-0.out‚25485.583984375 disk.sda1.out‚0 disk.rbd1.out‚ 14.3779296875" disk.nvme0c0n1.in‚ 370.58984375 disk.drbd2080.out‚0 network.sent‚ 1.4748443e+07 disk.drbd2080.in‚0# proc.memory.rssMB‚ 2693.87109375®ÿòv2ê- _wandbruntime‚4926  _runtime‚49262Zá Éëà trainer/global_step‚6149 % _timestamp‚1.7884003019741685e+09 _runtime‚4926.611022711 _step‚503 ' lr-AdamW/pg1‚0.00012297270353527239 ' lr-AdamW/pg2‚0.00012297270353527239KÌÎìÈ trainer/global_step‚6149 % _timestamp‚1.7884003019741685e+09 _runtime‚4926.611022711 _step‚503 ' lr-AdamW/pg1‚0.00012297270353527239 ' lr-AdamW/pg2‚0.00012297270353527239÷U>˜l2í- _wandbruntime‚4926  _runtime‚49265wÝ'î×O UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚173853301392642 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82463637 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 _step‚504 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82463099 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚182340672523776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚6149 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124435040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8829120120832 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8654045818494 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81235895 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚182340859526656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8679069668864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124435040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚173853483553406 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚173661771854848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚173661789857792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8678900668928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205690975 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47424254 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚192830300573696 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚173661789857792 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚165199420695552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124455078 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8678900668928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81235897 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8829332802560 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8653880697090 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35038845 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81235895 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚182340672523776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184000967771136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚173661771854848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.788400301974902e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205670935 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124455078 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚192830063951872 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚182340859526656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47424256 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _runtime‚ 4926.61192835 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81235897 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35039381 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184000943831040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0í¹«Ðâ'ïÜO WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8678900668928 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚81235897 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8829332802560 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8653880697090 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35038845 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81235895 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚182340672523776 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚184000967771136 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚173661771854848 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 $ _timestamp‚1.788400301974902e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 205670935 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 124455078 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚192830063951872 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚182340859526656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47424256 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _runtime‚ 4926.61192835 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81235897 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35039381 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚184000943831040 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚173853301392642 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚82463637 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 _step‚504 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚82463099 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚182340672523776 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 trainer/global_step‚6149 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 124435040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8829120120832 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8654045818494 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81235895 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚182340859526656 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8679069668864 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 124435040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚173853483553406 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚173661771854848 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚173661789857792 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8678900668928 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 205690975 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47424254 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚192830300573696 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚173661789857792 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚165199420695552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 124455078 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0øz ˜2ð- _wandbruntime‚4927  _runtime‚4927ÀÕ7à\ñãM ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8830536158720 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚165225363877888 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚165225380917248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82475729 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205703381 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 % _timestamp‚1.7884003026556978e+09 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8680462820352 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205723447 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124454827 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173689041219584 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124474891 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81248554 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 trainer/global_step‚6149 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚182369335026688 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚173880815900494 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8655434983246 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124454827 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚173689059222528 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184029880719360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8655269861634 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚182369522042880 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚173689041219584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82476266 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205703381 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205723447 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81248556 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35044633 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8830750924288 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚505 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚81248554 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚173689059222528 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8680293807104 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚184029856779264 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚173880633739522 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚182369522042880 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚82475729 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 205703381 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 % _timestamp‚1.7884003026556978e+09 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8680462820352 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 205723447 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 124454827 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚173689041219584 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 124474891 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81248554 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 trainer/global_step‚6149 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚182369335026688 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚173880815900494 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8655434983246 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 124454827 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚173689059222528 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚184029880719360 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8655269861634 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚182369522042880 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚173689041219584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚82476266 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 205703381 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 205723447 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81248556 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35044633 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8830750924288 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0ùj’Ò2ó- _wandbruntime‚4927  _runtime‚4927Ô„d\ôÖ /loss/FRt2/assignment_loss‚4.125602722167969 /loss/FRt4/detection_loss‚1.8614313890166738 1loss/SRqqt4/detection_loss‚1.2315987420221002 /loss/FBt3/detection_loss‚3.1578994035720824 &loss/total_loss‚47.449277303940846 2loss/SRqqt1/assignment_loss‚1.3577521966287383 /loss/FRt3/detection_loss‚2.7656419719968524 1loss/SRqqt1/detection_loss‚1.6394787160334763 /loss/FBt4/detection_loss‚0.6964702811217993 0loss/FRt3/assignment_loss‚3.4721952795982363 /loss/FRt1/detection_loss‚2.0528647789701084 /loss/FBt2/detection_loss‚3.1736147329211235 _runtime‚4927.294025739 /loss/FRt2/detection_loss‚2.8261532429605722 /loss/FBt1/assignment_loss‚0.210673087509349 1loss/SRqqt3/detection_loss‚1.7923898817376887 _step‚506 0loss/FRt4/assignment_loss‚1.5493536616893524 1loss/SRqqt2/detection_loss‚2.9364627255843234 trainer/global_step‚6149 2loss/SRqqt2/assignment_loss‚1.9615459081072073 2loss/SRqqt3/assignment_loss‚1.8716893319753891 2loss/SRqqt4/assignment_loss‚1.0174506754630197 .loss/FBt1/detection_loss‚4.069055154919624 % _timestamp‚1.7884003026572943e+09 /loss/FRt1/assignment_loss‚3.073969850774671 1loss/FBt2/assignment_loss‚0.26439658319239356 0loss/FBt3/assignment_loss‚0.2370476856827736 /loss/FBt4/assignment_loss‚0.104539300295319 epoch‚5ÞúååaõÛ 0loss/FRt4/assignment_loss‚1.5493536616893524 1loss/SRqqt2/detection_loss‚2.9364627255843234 trainer/global_step‚6149 2loss/SRqqt2/assignment_loss‚1.9615459081072073 2loss/SRqqt3/assignment_loss‚1.8716893319753891 2loss/SRqqt4/assignment_loss‚1.0174506754630197 .loss/FBt1/detection_loss‚4.069055154919624 % _timestamp‚1.7884003026572943e+09 /loss/FRt1/assignment_loss‚3.073969850774671 1loss/FBt2/assignment_loss‚0.26439658319239356 0loss/FBt3/assignment_loss‚0.2370476856827736 /loss/FBt4/assignment_loss‚0.104539300295319 epoch‚5 /loss/FRt2/assignment_loss‚4.125602722167969 /loss/FRt4/detection_loss‚1.8614313890166738 1loss/SRqqt4/detection_loss‚1.2315987420221002 /loss/FBt3/detection_loss‚3.1578994035720824 &loss/total_loss‚47.449277303940846 2loss/SRqqt1/assignment_loss‚1.3577521966287383 /loss/FRt3/detection_loss‚2.7656419719968524 1loss/SRqqt1/detection_loss‚1.6394787160334763 /loss/FBt4/detection_loss‚0.6964702811217993 0loss/FRt3/assignment_loss‚3.4721952795982363 /loss/FRt1/detection_loss‚2.0528647789701084 /loss/FBt2/detection_loss‚3.1736147329211235 _runtime‚4927.294025739 /loss/FRt2/detection_loss‚2.8261532429605722 /loss/FBt1/assignment_loss‚0.210673087509349 1loss/SRqqt3/detection_loss‚1.7923898817376887 _step‚506ú¬KÉy‘ö‚(:† ¶¥ãÔä’ÿ­ gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚139.685' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.937 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0†:V‚÷‚(:÷ ¶¥ãÔÇýDZ disk.dm-1.out‚ 7616.13671875" disk.dm-0.out‚25490.5400390625 network.recv‚ 1.784646e+06% disk./.usageGB‚226.83970260620117 disk.rbd1.out‚ 14.3779296875* proc.memory.percent‚16.442084312438965 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 370.58984375 proc.cpu.threads‚17 network.sent‚ 1.4753819e+07* disk./.usagePercent‚15.223402773089994 disk.drbd2080.in‚0 disk.dm-0.in‚ 370.58984375 disk.sda3.in‚ 0.0546875$ memory_percent‚94.51384544372559' proc.memory.availableMB‚ 898.8515625' disk.nvme0c0n1.out‚25490.5400390625 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2693.87109375 disk.sda3.out‚ 7616.13671875 disk.sda.out‚ 7616.13671875 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 cpu‚25.376857080828366 disk.sda.in‚ 0.0546875 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 370.58984375% disk.nvme0n1.out‚25490.5400390625 disk.dm-1.in‚ 0.0546875ÕI}ñø‚(:’ Å¥ãÔ–ê® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚147.246' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.449200000000005 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0å¾…ù‚(:ú Å¥ãÔ¾÷± disk.sda2.out‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 14.3779296875 disk.sda.in‚ 0.0546875 disk.dm-1.in‚ 0.0546875 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 370.58984375( disk.nvme0c0n1.out‚25493.43701171875 network.sent‚ 1.4862361e+07* disk./.usagePercent‚15.223402773089994 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0& disk.nvme0n1.out‚25493.43701171875 disk.drbd1716.in‚0 disk.drbd2265.out‚0# disk.dm-0.out‚25493.43701171875$ memory_percent‚94.51470375061035' proc.memory.availableMB‚ 898.7109375* proc.memory.percent‚16.442084312438965 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.dm-0.in‚ 370.58984375 disk.sda3.in‚ 0.0546875# proc.memory.rssMB‚ 2693.87109375 disk.sda.out‚ 7616.71484375 disk.dm-1.out‚ 7616.71484375% disk./.usageGB‚226.83970260620117 disk.rbd0.in‚0 disk.sda3.out‚ 7616.71484375 network.recv‚ 1.790572e+06 disk.sda1.out‚0 disk.nvme0n1.in‚ 370.58984375 cpu‚25.375877271635414 proc.cpu.threads‚17,ÍXÑ2ú- _wandbruntime‚4962  _runtime‚4962‰”öÉûà ' lr-AdamW/pg1‚0.00012143552955050266 ' lr-AdamW/pg2‚0.00012143552955050266 trainer/global_step‚6199 % _timestamp‚1.7884003376291091e+09 _runtime‚4962.265973541 _step‚507M9 LÎüÈ ' lr-AdamW/pg1‚0.00012143552955050266 ' lr-AdamW/pg2‚0.00012143552955050266 trainer/global_step‚6199 % _timestamp‚1.7884003376291091e+09 _runtime‚4962.265973541 _step‚507û× 2Ë2ý- _wandbruntime‚4962  _runtime‚4962 Û‘šß'þÙO XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47793106 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207296587 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81868845 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚175220114488614 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚175025240091648 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8748571991552 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83100479 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚185446591242240 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8900007520256 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8748740991488 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183773812083200 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚194346598762496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚183773999086080 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8723352515498 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8900220201984 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81868847 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125427742 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚4962.266853912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚166496579812352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47793104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚185446615182336 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125427742 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚175025258094592 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚175219932327850 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125447780 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚514 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8723517636902 % _timestamp‚1.7884003376298277e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚183773999086080 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207316627 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83100993 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81868845 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚24605J32 ÿÞO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚166496596851712 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚183773812083200 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _step‚508 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚512 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚81868847 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 125427742 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚4962.266853912 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚166496579812352 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚47793104 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚185446615182336 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_startnÒ†ÖÆ/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 125427742 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚175025258094592 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚175219932327850 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 125447780 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚514 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8723517636902 % _timestamp‚1.7884003376298277e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚183773999086080 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 207316627 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83100993 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚81868845 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚47793106 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 207296587 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚81868845 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚175220114488614 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚175025240091648 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8748571991552 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83100479 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚185446591242240 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8900007520256 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8748740991488 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚183773812083200 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚194346598762496 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚183773999086080 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8723352515498 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8900220201984 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207349099 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35313122 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚175052509456384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8750134142976 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47800483 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125467593 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚175052527459328 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8901422507520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚509 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚175247264674730 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚183802661602304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚175052527459328 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207329033 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81881506 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚194377163306496 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚183802474586112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83113081 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8749965129728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81881504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81881506 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚194376926697984 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚185475528130560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8901635175936 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8724906801654 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚183802661602304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83113605 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207329033 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚185475504190464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 125447529 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 % _timestamp‚1.7884003383007095e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8724741680042 trainer/global_step‚6199C*Xžî&‚èM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚183802474586112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83113081 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8749965129728 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚81881504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚81881506 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚194376926697984 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚185475528130560 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8901635175936 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8724906801654 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚183802661602304 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83113605 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 207329033 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚185475504190464 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 125447529 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 % _timestamp‚1.7884003383007095e+09 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8724741680042 trainer/global_step‚6199 _runtime‚4962.938139477 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 207349099 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚166522522994688 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 207349099 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35313122 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚175052509456384 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8750134142976 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚47800483 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 125467593 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚175052527459328 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 8901422507520 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚509 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚175247264674730 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚183802661602304 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚175052527459328 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 207329033 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚81881506 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚194377163306496ý z‹¿2ƒ- _wandbruntime‚4962  _runtime‚4962i©a?V„Ð 0loss/FBt3/assignment_loss‚0.4220827218766014 .loss/FBt1/detection_loss‚2.439067249496778 2loss/SRqqt1/assignment_loss‚1.5361823221402509 1loss/FBt2/assignment_loss‚0.44426173204517266 1loss/FBt4/assignment_loss‚0.16715376846098134 .loss/SRqqt2/detection_loss‚2.4083692916802 1loss/SRqqt4/detection_loss‚1.1004098067199117 .loss/FBt2/detection_loss‚2.552574172616005 trainer/global_step‚6199 /loss/FRt1/detection_loss‚2.0152717397131314 .loss/FRt3/detection_loss‚2.469616728524367 1loss/SRqqt3/detection_loss‚1.7314957340761108 .loss/FBt3/detection_loss‚4.022019828359286 epoch‚5 _step‚510 1loss/SRqqt2/assignment_loss‚2.289012991956302 /loss/FBt1/assignment_loss‚0.332682907639537 /loss/FRt4/detection_loss‚1.6153986201274628 % _timestamp‚1.7884003383024101e+09 _runtime‚4962.939181103 /loss/FRt2/assignment_loss‚3.694825764360099 0loss/FRt3/assignment_loss‚3.2589031594494977 /loss/FRt2/detection_loss‚3.0021239498565935 1loss/SRqqt1/detection_loss‚1.6722236888749258 0loss/FRt1/assignment_loss‚3.1628013828705095 .loss/FBt4/detection_loss‚0.481517760212118 2loss/SRqqt4/assignment_loss‚1.2754244404052621 /loss/FRt4/assignment_loss‚1.431168364543541 2loss/SRqqt3/assignment_loss‚1.6057869807189824 &loss/total_loss‚45.130375106723626\é‹ú[…Õ 2loss/SRqqt4/assignment_loss‚1.2754244404052621 /loss/FRt4/assignment_loss‚1.431168364543541 2loss/SRqqt3/assignment_loss‚1.6057869807189824 &loss/total_loss‚45.130375106723626 0loss/FBt3/assignment_loss‚0.4220827218766014 .loss/FBt1/detection_loss‚2.439067249496778 2loss/SRqqt1/assignment_loss‚1.5361823221402509 1loss/FBt2/assignment_loss‚0.44426173204517266 1loss/FBt4/assignment_loss‚0.16715376846098134 .loss/SRqqt2/detection_loss‚2.4083692916802 1loss/SRqqt4/detection_loss‚1.1004098067199117 .loss/FBt2/detection_loss‚2.552574172616005 trainer/global_step‚6199 /loss/FRt1/detection_loss‚2.0152717397131314 .loss/FRt3/detection_loss‚2.469616728524367 1loss/SRqqt3/detection_loss‚1.7314957340761108 .loss/FBt3/detection_loss‚4.022019828359286 epoch‚5 _step‚510 1loss/SRqqt2/assignment_loss‚2.289012991956302 /loss/FBt1/assignment_loss‚0.332682907639537 /loss/FRt4/detection_loss‚1.6153986201274628 % _timestamp‚1.7884003383024101e+09 _runtime‚4962.939181103 /loss/FRt2/assignment_loss‚3.694825764360099 0loss/FRt3/assignment_loss‚3.2589031594494977 /loss/FRt2/detection_loss‚3.0021239498565935 1loss/SRqqt1/detection_loss‚1.6722236888749258 0loss/FRt1/assignment_loss‚3.1628013828705095 .loss/FBt4/detection_loss‚0.481517760212118þ|"GŒ‘†‚(:† Ô¥ãÔ„¿»® gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚124.865' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.973 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£Uýsj‡‚(:ß Ô¥ãÔ±û± disk.nvme0n1.in‚ 370.58984375 disk.drbd2266.in‚0 disk.drbd2080.out‚0 network.sent‚ 1.4868413e+07 disk.sda3.out‚ 7617.171875 disk.rbd1.in‚0$ memory_percent‚94.52087879180908* disk./.usagePercent‚15.223680285257114$ disk./.usageGB‚226.8438377380371 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.dm-0.in‚ 370.58984375 disk.sda1.in‚0# proc.memory.rssMB‚ 2693.87109375 disk.sda.in‚ 0.0546875 disk.dm-0.out‚ 25497.3125 disk.sda1.out‚0 disk.drbd2266.out‚0 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.0546875 disk.drbd2080.in‚0 disk.sda3.in‚ 0.0546875! disk.nvme0c0n1.out‚ 25497.3125* proc.memory.percent‚16.442084312438965 proc.cpu.threads‚17 disk.sda.out‚ 7617.171875 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 370.58984375( proc.memory.availableMB‚ 897.69921875 network.recv‚ 1.794597e+06 disk.sda2.in‚0 disk.dm-1.out‚ 7617.171875 disk.rbd1.out‚ 14.3779296875 disk.nvme0n1.out‚ 25497.3125 cpu‚25.374887662592986ë´™ˆ‚(:… ã¥ãÔÓÞû­ gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚35.0 gpu.0.powerWatts‚137.96' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.592 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚00,ÄDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8970843357696 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚176586745423822 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83739932 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221070336 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126440482 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚176388726331392 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82501795 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8818412314112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185206951642624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35577976 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚186892238653440 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8818412314112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195863327021568 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚185206951642624 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245010432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚167793755968512 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195863082011136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126420444 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48161956 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82501797 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35577442 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚176388726331392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚167793738929152 _step‚512 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185207138645504 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185207138645504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48161954 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8818243314176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8818243314176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82501795 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚6249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8792989455310 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 208942279 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126440482E‘lžã'‘ÝO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 208922239 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚83739396 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8792824333906 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚185206951642624 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35577976 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚186892238653440 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8818412314112 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚195863327021568 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚185206951642624 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245010432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚167793755968512 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚195863082011136 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 126420444 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48161956 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚82501797 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35577442 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚176388726331392 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚167793738929152 _step‚512 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚185207138645504 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚185207138645504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48161954 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8818243314176 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8818243314176 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚82501795 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚6249 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8792989455310 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 208942279 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 126440482 % _timestamp‚1.7884003733328774e+09 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚186892262593536 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 8971064428032 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚ 4997.96991258 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚176388708328448 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚176388708328448 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 8970843357696 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚176586745423822 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚83739932 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221070336 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 126440482 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚176388726331392 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚82501795 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8818412314112 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185235801161728 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48169331 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126460295 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚511 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 208974751 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82514454 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185235614145536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8819636452352 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚ 4998.64282448 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208954685 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8819805465600 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 208954685 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83752686 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8972476028928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82514456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35583864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _step‚513 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82514456 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82514454 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8819805465600 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚176415977693184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83753197 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚176415995696128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8794213498450 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚176415977693184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚167819699150848 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195893410767872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚509 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208974751 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460ˆzUbì&”æM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚195893651570688 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6249 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚185235801161728 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48169331 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 126460295 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚511 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 208974751 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚82514454 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚185235614145536 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8819636452352 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚ 4998.64282448 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 208954685 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8819805465600 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 208954685 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚83752686 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 8972476028928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚82514456 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35583864 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _step‚513 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚82514456 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚82514454 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8819805465600 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚176415977693184 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚83753197 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚176415995696128 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8794213498450 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚176415977693184 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚167819699150848 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚195893410767872 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚509 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 208974751 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚176613895609938 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚186921151601664 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚176614077770910 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚185235801161728X’KÎ2•- _wandbruntime‚4998  _runtime‚4998÷æ Ð^–Ø 0loss/FRt4/assignment_loss‚2.4915854971287614 1loss/FBt2/assignment_loss‚0.26556760910228727 1loss/SRqqt2/detection_loss‚2.4518377065658568 /loss/FRt2/detection_loss‚3.4065948222812854 1loss/SRqqt3/assignment_loss‚1.397821265201156 1loss/SRqqt1/detection_loss‚1.8886476755142212 trainer/global_step‚6249 _step‚514 1loss/SRqqt3/detection_loss‚1.7965406305514848 2loss/FBt3/assignment_loss‚0.006673717402884449 1loss/SRqqt2/assignment_loss‚2.130316518034254 1loss/SRqqt4/detection_loss‚1.0624136595870368 2loss/SRqqt1/assignment_loss‚1.6445167576711552 1loss/FBt4/assignment_loss‚0.20964505099836941 /loss/FRt4/detection_loss‚1.8662888371085717 .loss/FBt2/detection_loss‚2.664694320071827 /loss/FBt4/detection_loss‚0.7732224701156403 epoch‚5 2loss/SRqqt4/assignment_loss‚1.2218011794393533 0loss/FBt1/assignment_loss‚0.3039850761803488 /loss/FRt3/detection_loss‚2.5087159752845762 /loss/FBt1/detection_loss‚2.3541113336881003 .loss/FBt3/detection_loss‚2.713003212213516 % _timestamp‚1.7884003740072386e+09 /loss/FRt2/assignment_loss‚4.292339644934001 %loss/total_loss‚45.56089063447489 _runtime‚4998.644003114 /loss/FRt1/detection_loss‚2.1223434439876625 0loss/FRt1/assignment_loss‚2.7948315582777323 0loss/FRt3/assignment_loss‚3.1933926731348037©мc—Ý /loss/FRt1/detection_loss‚2.1223434439876625 0loss/FRt1/assignment_loss‚2.7948315582777323 0loss/FRt3/assignment_loss‚3.1933926731348037 0loss/FRt4/assignment_loss‚2.4915854971287614 1loss/FBt2/assignment_loss‚0.26556760910228727 1loss/SRqqt2/detection_loss‚2.4518377065658568 /loss/FRt2/detection_loss‚3.4065948222812854 1loss/SRqqt3/assignment_loss‚1.397821265201156 1loss/SRqqt1/detection_loss‚1.8886476755142212 trainer/global_step‚6249 _step‚514 1loss/SRqqt3/detection_loss‚1.7965406305514848 2loss/FBt3/assignment_loss‚0.006673717402884449 1loss/SRqqt2/assignment_loss‚2.130316518034254 1loss/SRqqt4/detection_loss‚1.0624136595870368 2loss/SRqqt1/assignment_loss‚1.6445167576711552 1loss/FBt4/assignment_loss‚0.20964505099836941 /loss/FRt4/detection_loss‚1.8662888371085717 .loss/FBt2/detection_loss‚2.664694320071827 /loss/FBt4/detection_loss‚0.7732224701156403 epoch‚5 2loss/SRqqt4/assignment_loss‚1.2218011794393533 0loss/FBt1/assignment_loss‚0.3039850761803488 /loss/FRt3/detection_loss‚2.5087159752845762 /loss/FBt1/detection_loss‚2.3541113336881003 .loss/FBt3/detection_loss‚2.713003212213516 % _timestamp‚1.7884003740072386e+09 /loss/FRt2/assignment_loss‚4.292339644934001 %loss/total_loss‚45.56089063447489 _runtime‚4998.644003114‚£BªÃ˜‚(:’ ¦ãÔ­î­¯ gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚123.221' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.644199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™rWT~™‚(:ó ¦ãÔýǾ± cpu‚25.371988870903902 disk.sda.in‚ 0.0546875 disk.dm-1.in‚ 0.0546875 disk.rbd0.out‚0" disk.dm-0.out‚25513.4384765625 disk.sda1.out‚0 disk.rbd1.out‚ 14.3779296875* disk./.usagePercent‚15.223548953392047% disk./.usageGB‚226.84188079833984 disk.drbd1716.in‚0" disk.nvme0c0n1.in‚ 370.60546875 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda3.in‚ 0.0546875' disk.nvme0c0n1.out‚25513.4384765625 disk.nvme0n1.in‚ 370.60546875& proc.memory.availableMB‚ 897.234375 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0# proc.memory.rssMB‚ 2693.87109375 disk.sda1.in‚0 disk.drbd2266.out‚0 network.sent‚ 1.4988619e+07 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 370.60546875% disk.nvme0n1.out‚25513.4384765625 disk.sda.out‚ 7621.3984375 disk.drbd2080.out‚0$ memory_percent‚94.52371597290039* proc.memory.percent‚16.442084312438965 network.recv‚ 1.809107e+06 disk.dm-1.out‚ 7621.3984375 disk.drbd2265.in‚0 disk.sda3.out‚ 7621.3984375ð\»ºš‚(:’ ¦ãÔˆŒ® gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚142.932' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.586399999999994 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Cñÿm›‚(:â ¦ãÔï¬Ý± disk.sda2.out‚0 disk.dm-1.out‚ 7622.046875 disk.dm-0.out‚ 25513.890625 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚94.52524185180664& proc.memory.availableMB‚ 896.984375 proc.cpu.threads‚17 network.sent‚ 1.5096409e+07 disk.sda2.in‚0 disk.drbd1716.out‚0) disk./.usagePercent‚15.22354920939958 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda3.out‚ 7622.046875 disk.rbd1.in‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.dm-1.in‚ 0.0546875* proc.memory.percent‚16.442084312438965 cpu‚25.371030179212212 network.recv‚ 1.814428e+06 disk.drbd2265.in‚0 disk.sda.out‚ 7622.046875 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚ 0.0546875" disk.nvme0c0n1.in‚ 370.60546875$ disk./.usageGB‚226.8418846130371 disk.nvme0n1.in‚ 370.60546875# proc.memory.rssMB‚ 2693.87109375 disk.drbd2080.in‚0 disk.dm-0.in‚ 370.60546875 disk.rbd1.out‚ 14.3779296875# disk.nvme0c0n1.out‚ 25513.890625! disk.nvme0n1.out‚ 25513.890625 disk.sda.in‚ 0.0546875Ú†øü2œ- _wandbruntime‚5033  _runtime‚5033sÔÓÇÁ & lr-AdamW/pg1‚0.0001183527859162677 & lr-AdamW/pg2‚0.0001183527859162677 trainer/global_step‚6299 % _timestamp‚1.7884004088756404e+09 _runtime‚5033.512526099 _step‚515P݉oÌžÆ _step‚515 & lr-AdamW/pg1‚0.0001183527859162677 & lr-AdamW/pg2‚0.0001183527859162677 trainer/global_step‚6299 % _timestamp‚1.7884004088756404e+09 _runtime‚5033.512526099ƒ®ña¶2Ÿ- _wandbruntime‚5033  _runtime‚5033ô7;ß' ÙO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚188337910004736 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9041977222656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48530804 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35846569 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚177953376359030 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35847099 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8888083636736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚188337886064640 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 210567931 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84377373 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127433184 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8862461273718 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127413146 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83134745 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚186640091202048 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚186640091202048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚177752194568192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884004088763955e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83134745 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48530806 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚177953194198266 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚177752176565248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8887914636800 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83134747 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚186640278204928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210547891 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚186640278204928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8888083636736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚197379887227392 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚177752176565248 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8862296152314 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚5033.513429825 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000¿ª¼~¡ÞO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚188337886064640 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 210567931 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚84377373 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 127433184 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 8862461273718 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 127413146 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚83134745 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚186640091202048 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚186640091202048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚177752194568192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884004088763955e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚83134745 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 D:DeviceStatsMoni‡\fftor.on_train_batch_start/active.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚48530806 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚177953194198266 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚177752176565248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8887914636800 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚83134747 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚186640278204928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 210547891 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚186640278204928 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 8888083636736 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚197379887227392 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚177752176565248 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 8862296152314 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚5033.513429825 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚188337910004736 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9041977222656 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚48530804 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚35846569 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚177953376359030 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚35847099 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8888083636736 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038„ÚT¼Å2¢- _wandbruntime‚5034  _runtime‚5034šžhFé&£ãM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210600403 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8889476788224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48538181 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚169116858267648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 trainer/global_step‚6299 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127432933 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84390698 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9043179386880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84390153 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚197409978399744 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127452997 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚177779445929984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 210580337 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35852515 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83147406 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5034.191730703 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚197410210813952 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚169116841228288 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48538183 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83147404 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8889307774976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83147404 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35851972 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8889307774976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚188366799012864 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9043387860992?a ûî&¤èM UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8889307774976 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚188366799012864 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9043387860992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 210600403 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8889476788224 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚48538181 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚169116858267648 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 trainer/global_step‚6299 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 127432933 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚84390698 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9043179386880 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚84390153 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚197409978399744 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 127452997 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚177779445929984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 210580337 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚35852515 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚83147406 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚5034.191730703 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚197410210813952 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚169116841228288 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48538183 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83147404 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8889307774976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚83147404 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚35851972 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160…!•Öß2¥- _wandbruntime‚5034  _runtime‚5034ˆɲ`¦Ú 0loss/FRt3/assignment_loss‚3.2596416413784026 2loss/SRqqt4/assignment_loss‚0.9492064483818554 1loss/SRqqt2/detection_loss‚2.5407987011201456 /loss/FBt2/detection_loss‚2.7568404227495193 2loss/SRqqt2/assignment_loss‚2.2091511325402693 1loss/SRqqt1/detection_loss‚1.8148169014251456 trainer/global_step‚6299 _runtime‚5034.192833824 epoch‚5 $ _timestamp‚1.788400409556105e+09 /loss/FRt2/assignment_loss‚4.235960652751308 2loss/SRqqt1/assignment_loss‚1.6958990475900892 2loss/SRqqt3/assignment_loss‚1.6495772503620507 0loss/FBt3/assignment_loss‚0.6461356356739998 &loss/total_loss‚47.976896903719464 0loss/FBt4/assignment_loss‚0.1641215407866715 /loss/FRt4/detection_loss‚1.5666660319461867 1loss/SRqqt4/detection_loss‚1.1221936432615158 /loss/FBt1/detection_loss‚3.7859504342079164 _step‚518 1loss/FBt2/assignment_loss‚0.19018057672303712 /loss/FBt3/detection_loss‚3.7554577738046646 0loss/FRt1/assignment_loss‚3.4534591573183655 0loss/FRt4/assignment_loss‚1.4940042937243427 1loss/FBt1/assignment_loss‚0.24992563308260288 /loss/FRt2/detection_loss‚2.8942179122278766 /loss/FRt3/detection_loss‚2.8846698264280954 /loss/FRt1/detection_loss‚2.0741915136087137 1loss/SRqqt3/detection_loss‚1.8596377476137511 /loss/FBt4/detection_loss‚0.72419298501293840€e§ß 0loss/FRt3/assignment_loss‚3.2596416413784026 2loss/SRqqt4/assignment_loss‚0.9492064483818554 1loss/SRqqt2/detection_loss‚2.5407987011201456 /loss/FBt2/detection_loss‚2.7568404227495193 2loss/SRqqt2/assignment_loss‚2.2091511325402693 1loss/SRqqt1/detection_loss‚1.8148169014251456 trainer/global_step‚6299 _runtime‚5034.192833824 epoch‚5 $ _timestamp‚1.788400409556105e+09 /loss/FRt2/assignment_loss‚4.235960652751308 2loss/SRqqt1/assignment_loss‚1.6958990475900892 2loss/SRqqt3/assignment_loss‚1.6495772503620507 0loss/FBt3/assignment_loss‚0.6461356356739998 &loss/total_loss‚47.976896903719464 0loss/FBt4/assignment_loss‚0.1641215407866715 /loss/FRt4/detection_loss‚1.5666660319461867 1loss/SRqqt4/detection_loss‚1.1221936432615158 /loss/FBt1/detection_loss‚3.7859504342079164 _step‚518 1loss/FBt2/assignment_loss‚0.19018057672303712 /loss/FBt3/detection_loss‚3.7554577738046646 0loss/FRt1/assignment_loss‚3.4534591573183655 0loss/FRt4/assignment_loss‚1.4940042937243427 1loss/FBt1/assignment_loss‚0.24992563308260288 /loss/FRt2/detection_loss‚2.8942179122278766 /loss/FRt3/detection_loss‚2.8846698264280954 /loss/FRt1/detection_loss‚2.0741915136087137 1loss/SRqqt3/detection_loss‚1.8596377476137511 /loss/FBt4/detection_loss‚0.7241929850129384†ü ÕÓ¨‚(:… Ÿ¦ãÔÈБ® gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚146.42' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0X´’…©‚(:ú Ÿ¦ãÔÂ÷ú° disk.drbd2080.in‚0 disk.sda3.out‚ 7622.58984375 disk.drbd1716.in‚0 disk.dm-1.out‚ 7622.58984375 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0# disk.dm-0.out‚27159.88134765625* proc.memory.percent‚16.442084312438965( disk.nvme0c0n1.out‚27159.88134765625 disk.rbd0.in‚0 disk.dm-0.in‚ 370.98828125# proc.memory.rssMB‚ 2693.87109375 cpu‚25.370087254909812 proc.cpu.threads‚17 network.sent‚ 1.5102663e+07 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 370.98828125 disk.sda.out‚ 7622.58984375 disk.sda2.out‚0% disk./.usageGB‚229.16759872436523 disk.sda.in‚ 0.0546875 disk.sda1.out‚0 network.recv‚ 1.818958e+06' proc.memory.availableMB‚ 896.8671875* disk./.usagePercent‚15.379629834814928 disk.nvme0n1.in‚ 370.98828125& disk.nvme0n1.out‚27159.88134765625 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 14.3779296875 disk.dm-1.in‚ 0.0546875 disk.rbd0.out‚0 disk.sda3.in‚ 0.0546875$ memory_percent‚94.52595710754395Ç NTª‚(:’ ®¦ãԾ딮 gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚141.288' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.257600000000004 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$ÈjVt«‚(:é ®¦ãÔÂÚ° disk.drbd2266.out‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 375.0546875 disk.drbd2080.in‚0 cpu‚25.369095604899847 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda.out‚ 7623.25390625 disk.drbd2080.out‚0 disk.dm-0.in‚ 375.0546875! disk.nvme0c0n1.in‚ 375.0546875$ disk.nvme0c0n1.out‚ 27724.3046875 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2693.87109375* proc.memory.percent‚16.442084312438965 disk.sda3.out‚ 7623.25390625 disk.sda.in‚ 0.0546875 disk.drbd2265.in‚0$ memory_percent‚94.52447891235352 proc.cpu.threads‚17% disk./.usageGB‚229.49512100219727 disk.sda3.in‚ 0.0546875 disk.rbd1.out‚ 14.3779296875 disk.sda2.out‚0 disk.dm-1.out‚ 7623.25390625 disk.drbd2265.out‚0 network.sent‚ 1.5210902e+07" disk.nvme0n1.out‚ 27724.3046875 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.rbd0.out‚0 network.recv‚ 1.824492e+06 disk.dm-0.out‚ 27724.3046875 disk.dm-1.in‚ 0.0546875& proc.memory.availableMB‚ 897.109375) disk./.usagePercent‚15.40161012968973\ÞÝÉ2¬- _wandbruntime‚5069  _runtime‚5069üýv¨È­Â $ _timestamp‚1.788400445016697e+09 _runtime‚5069.653553492 _step‚519 ' lr-AdamW/pg1‚0.00011680785870595532 ' lr-AdamW/pg2‚0.00011680785870595532 trainer/global_step‚6349Q|Ãꨂ trainer/global_step‚6349 $ _timestamp‚1.788400445016697e+09 _runtime‚5069.653553492 _step‚519 ' lr-AdamW/pg1‚0.00011680785870595532 ' lr-AdamW/pg2‚0.00011680785870595532‡(©Dƒ2¯- _wandbruntime‚5069  _runtime‚5069ñeqH °ØO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 212173543 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _runtime‚5069.654451767 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚188073230761472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚179319825133474 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_¨ ½ËÓ"batch_start/active.small_pool.allocated‚ 128425886 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚189783533475840 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8957585959424 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8957585959424 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚179115662804992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚170388074202112 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128405848 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85017460 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85016925 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚170388057162752 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚179115644802048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8957754959360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198896141261312 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚179115662804992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 212193583 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36117804 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128425886 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83767697 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚179115644802048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128405848 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 trainer/global_step‚6349 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚188073230761472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0„Ñ‚ã'±ÝO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 128405848 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85017460 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚85016925 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚170388057162752 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚179115644802048 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 8957754959360 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚198896141261312 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚179115662804992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 212193583 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36117804 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 128425886 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚83767697 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚179115644802048 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 128405848 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 trainer/global_step‚6349 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚188073230761472 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 212173543 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _runtime‚5069.654451767 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚188073230761472 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚179319825133474 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 128425886 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚189783533475840 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 8957585959424 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 8957585959424 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚179115662804992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚170388074202112ˆL=(Z’²‚(:‡ ½¦ãÔ¡Ÿš® gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚140.751' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.1502 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0(ñÀu³‚(:ê ½¦ãÔÛÚѱ disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.sda3.out‚ 7623.7421875% disk.nvme0n1.out‚28234.1259765625* proc.memory.percent‚16.442084312438965" disk.dm-0.out‚28234.1259765625 disk.sda1.out‚0 disk.sda.in‚ 0.0546875 network.recv‚ 1.828614e+06 disk.drbd1716.out‚0 disk.sda3.in‚ 0.0546875 disk.sda.out‚ 7623.7421875# proc.memory.rssMB‚ 2693.87109375$ disk./.usageGB‚229.4939079284668 disk.dm-0.in‚ 378.859375 disk.rbd1.out‚ 14.3779296875 disk.dm-1.out‚ 7623.7421875 disk.sda2.out‚0 disk.dm-1.in‚ 0.0546875 disk.rbd0.in‚0 disk.nvme0n1.in‚ 378.859375' proc.memory.availableMB‚ 897.1171875 cpu‚25.36819562160577 network.sent‚ 1.521688e+07 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.sda1.in‚0# memory_percent‚94.5244312286377 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 378.859375' disk.nvme0c0n1.out‚28234.1259765625* disk./.usagePercent‚15.401528719293841 disk.drbd2266.out‚0°ˆ$÷2´- _wandbruntime‚5070  _runtime‚5070+¬}é&µãM S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚188101893264384 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85030128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 8933157135266 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212205989 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8959148110848 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9114235129856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚179347157480354 % _timestamp‚1.7884004457034366e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83780354 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36122569 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚198926705494016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚179347339641326 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83780356 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚179142932169728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚189812446424064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 128445699 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚48907033 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8958979097600 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85029600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8958979097600 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚179142914166784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9114024558592 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚179142932169728 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189812470364160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212226055 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128445699 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212205989 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212226055 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128425635 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _step‚521 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚188102080280576 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚179142914166784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198926470982656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8933322256878 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8959148110848 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 8958979097600 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85029600 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 8958979097600 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚179142914166784 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9114024558592 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚179142932169728 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚189812470364160 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 212226055 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADevicœ»kù÷eStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 128445699 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 212205989 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 212226055 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 128425635 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _step‚521 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚188102080280576 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚179142914166784 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚198926470982656 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 8933322256878 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 8959148110848 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 212205989 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 8959148110848 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9114235129856 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚179347157480354 % _timestamp‚1.7884004457034366e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚83780354 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36122569 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚198926705494016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚179347339641326 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚83780356‰Äÿ½½2·- _wandbruntime‚5070  _runtime‚5070 ]G]¸× /loss/FRt2/assignment_loss‚4.254132861676424 1loss/FBt1/assignment_loss‚0.45970235472278936 0loss/SRqqt2/detection_loss‚2.687338424416689 1loss/SRqqt3/detection_loss‚1.8077484285622312 2loss/SRqqt3/assignment_loss‚1.5902070580622214 /loss/FRt1/detection_loss‚2.1032842192798853 epoch‚5 trainer/global_step‚6349 _runtime‚5070.341799262 /loss/FRt3/assignment_loss‚2.871430776621166 /loss/FBt1/detection_loss‚2.6125557592936923 /loss/FBt4/detection_loss‚0.8863774596498563 0loss/FRt1/assignment_loss‚3.4534046202898026 0loss/FRt4/assignment_loss‚0.9736008375883103 1loss/FBt3/assignment_loss‚0.20649914606110542 /loss/FRt4/detection_loss‚1.9685122158378363 .loss/FBt3/detection_loss‚2.332791857421398 %loss/total_loss‚45.33491135046689 2loss/SRqqt1/assignment_loss‚1.7714443960650401 1loss/FBt2/assignment_loss‚0.43362333357765115 /loss/FRt3/detection_loss‚2.4707098689518476 1loss/SRqqt1/detection_loss‚1.8046512693166732 2loss/SRqqt4/assignment_loss‚1.1331071145329121 1loss/FBt4/assignment_loss‚0.19614089439006988 0loss/SRqqt4/detection_loss‚1.045237178033268 /loss/FBt2/detection_loss‚2.3268080204725266 2loss/SRqqt2/assignment_loss‚2.7282029069386997 .loss/FRt2/detection_loss‚3.217400348704794 _step‚522 % _timestamp‚1.7884004457050736e+09-V!÷b¹Ü 0loss/SRqqt4/detection_loss‚1.045237178033268 /loss/FBt2/detection_loss‚2.3268080204725266 2loss/SRqqt2/assignment_loss‚2.7282029069386997 .loss/FRt2/detection_loss‚3.217400348704794 _step‚522 % _timestamp‚1.7884004457050736e+09 /loss/FRt2/assignment_loss‚4.254132861676424 1loss/FBt1/assignment_loss‚0.45970235472278936 0loss/SRqqt2/detection_loss‚2.687338424416689 1loss/SRqqt3/detection_loss‚1.8077484285622312 2loss/SRqqt3/assignment_loss‚1.5902070580622214 /loss/FRt1/detection_loss‚2.1032842192798853 epoch‚5 trainer/global_step‚6349 _runtime‚5070.341799262 /loss/FRt3/assignment_loss‚2.871430776621166 /loss/FBt1/detection_loss‚2.6125557592936923 /loss/FBt4/detection_loss‚0.8863774596498563 0loss/FRt1/assignment_loss‚3.4534046202898026 0loss/FRt4/assignment_loss‚0.9736008375883103 1loss/FBt3/assignment_loss‚0.20649914606110542 /loss/FRt4/detection_loss‚1.9685122158378363 .loss/FBt3/detection_loss‚2.332791857421398 %loss/total_loss‚45.33491135046689 2loss/SRqqt1/assignment_loss‚1.7714443960650401 1loss/FBt2/assignment_loss‚0.43362333357765115 /loss/FRt3/detection_loss‚2.4707098689518476 1loss/SRqqt1/detection_loss‚1.8046512693166732 2loss/SRqqt4/assignment_loss‚1.1331071145329121 1loss/FBt4/assignment_loss‚0.19614089439006988Š>e3º‚(:„ ̦ãÔñí¢® gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚35.0 gpu.0.powerWatts‚146.95' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.39 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¬Šâ»‚(:ö ̦ãÔòð" disk.nvme0c0n1.in‚ 378.94140625% disk.nvme0n1.out‚28235.3544921875* proc.memory.percent‚16.442084312438965 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 14.3779296875 disk.nvme0n1.in‚ 378.94140625 disk.sda.out‚ 7624.44140625' proc.memory.availableMB‚ 896.7890625 disk.drbd2080.in‚0 disk.sda3.in‚ 0.0546875 disk.sda2.out‚0 disk.dm-1.out‚ 7624.44140625 disk.drbd2266.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 378.94140625$ memory_percent‚94.52643394470215 cpu‚25.36717509912224 disk.rbd0.in‚0 disk.sda.in‚ 0.0546875 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0" disk.dm-0.out‚28235.3544921875 disk.sda2.in‚0 disk.drbd1716.out‚0% disk./.usageGB‚229.49388122558594 disk.sda1.out‚0 proc.cpu.threads‚17 network.recv‚ 1.834717e+06 disk.sda3.out‚ 7624.44140625' disk.nvme0c0n1.out‚28235.3544921875 disk.dm-1.in‚ 0.0546875# proc.memory.rssMB‚ 2693.87109375 network.sent‚ 1.5314186e+07* disk./.usagePercent‚15.401526927241102 disk.drbd2265.in‚0C …‘¼‚(:† Û¦ãÔŸ˜•® gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚122.395' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.479 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0–Õ¬}½‚(:ò Û¦ãÔôôÛ± disk.sda2.out‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0" disk.dm-0.out‚28235.6982421875 disk.sda1.out‚0( proc.memory.availableMB‚ 897.27734375 cpu‚25.366234565976097 disk.sda2.in‚0 disk.dm-0.in‚ 378.94140625 disk.rbd1.out‚ 14.3779296875% disk./.usageGB‚229.49334716796875 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 378.94140625' disk.nvme0c0n1.out‚28235.6982421875 network.sent‚ 1.5412465e+07 network.recv‚ 1.841204e+06 disk.sda.in‚ 0.0546875 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.sda3.out‚ 7624.984375% disk.nvme0n1.out‚28235.6982421875 disk.sda.out‚ 7624.984375 disk.dm-1.in‚ 0.0546875 disk.drbd2265.out‚0 disk.sda3.in‚ 0.0546875 disk.rbd1.in‚0* proc.memory.percent‚16.442084312438965 disk.nvme0n1.in‚ 378.94140625 disk.dm-1.out‚ 7624.984375 disk.rbd0.out‚0* disk./.usagePercent‚15.401491086186308 disk.drbd2080.out‚0$ memory_percent‚94.52345371246338# proc.memory.rssMB‚ 2693.87109375 proc.cpu.threads‚17áŒÎê2¾- _wandbruntime‚5104  _runtime‚5104?¶uªÉ¿Ã _runtime‚5104.884980327 _step‚523 ' lr-AdamW/pg1‚0.00011526099035699089 ' lr-AdamW/pg2‚0.00011526099035699089 trainer/global_step‚6399 % _timestamp‚1.7884004802481172e+09âp aÎÀÈ % _timestamp‚1.7884004802481172e+09 _runtime‚5104.884980327 _step‚523 ' lr-AdamW/pg1‚0.00011526099035699089 ' lr-AdamW/pg2‚0.00011526099035699089 trainer/global_step‚6399‹õ9ž,2Á- _wandbruntime‚5104  _runtime‚5104F¬ÛÞß'ÂÙO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9027257282048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚189506370320896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84400645 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 213819235 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚189506370320896 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129418588 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36389791 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129398550 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚180479131041792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 % _timestamp‚1.7884004802488356e+09 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 trainer/global_step‚6399 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚189506557323776 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200412701122048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚171685233318912 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚191229204827136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9183735013888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9183520235008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85658834 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚171685216279552\Âè¾ÃÞO eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚191229204827136 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9183735013888 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9183520235008 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚85658834 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚171685216279552 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9027257282048 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚189506370320896 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚84400645 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 213819235 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚189506370320896 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 129418588 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36389791 VHDev¼j9Q&iceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 129398550 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚180479131041792 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 % _timestamp‚1.7884004802488356e+09 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 trainer/global_step‚6399 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚189506557323776 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚200412701122048 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚171685233318912 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0Œ¹«Uá2Ä- _wandbruntime‚5105  _runtime‚5105 zè&ÅâM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚191258093835264 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9002628953674 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 129418337 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚180506382403584 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129438401 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85671573 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 213851707 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84413304 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49275883 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9184943121408 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 _step‚525 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚200443267273728 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚180713788415562 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49275881 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚180506400406528 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85671051 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚171711176501248 _runtime‚5105.559343073 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚189535219840000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚180713970576534 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚191258117775360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚180506382403584 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 213831641 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚171711159461888 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9028819433472 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129438401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84413306 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚85671051 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚171711176501248 _runtime‚5105.559343073 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚189535219840000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚180713970576534 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚191258117775360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚180506382403584 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 213831641 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚171711159461888 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9028819433472 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 129438401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚84413306 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚180506382403584 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 129438401 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚85671573 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 213851707 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚84413304 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49275883 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9184943121408 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 _step‚525ÍÜÌ«2Ç- _wandbruntime‚5105  _runtime‚5105J»‰\ÈÖ /loss/FRt1/detection_loss‚2.0135522155926147 .loss/FBt3/detection_loss‚2.243388129906221 0loss/FRt2/assignment_loss‚4.3427169487394135 .loss/FBt2/detection_loss‚3.405954910176141 /loss/FRt4/detection_loss‚1.6800163686275482 2loss/SRqqt1/assignment_loss‚1.7235411230409354 2loss/SRqqt3/assignment_loss‚1.4536480946426695 1loss/FBt1/assignment_loss‚0.23215110733639854 1loss/SRqqt4/detection_loss‚1.0391967262527564 .loss/FBt1/detection_loss‚2.240081305305163 1loss/FBt2/assignment_loss‚0.24876998524332908 1loss/SRqqt3/detection_loss‚2.0255441881240683 .loss/FRt3/detection_loss‚2.320101986080408 1loss/FBt4/assignment_loss‚0.12206977595022242 1loss/SRqqt1/detection_loss‚1.7595456875207132 1loss/SRqqt2/detection_loss‚2.4653382321883894 epoch‚5 _runtime‚5105.560301171 _step‚526 &loss/total_loss‚45.549494017483894 trainer/global_step‚6399 0loss/FRt1/assignment_loss‚3.3503850975940965 0loss/FRt3/assignment_loss‚2.9819231420755385 0loss/FRt4/assignment_loss‚1.9455816074023171 1loss/SRqqt2/assignment_loss‚2.188884316333409 -loss/FRt2/detection_loss‚2.97833835462044 /loss/FBt4/detection_loss‚1.3221336309798062 % _timestamp‚1.7884004809235444e+09 2loss/SRqqt4/assignment_loss‚0.9322130823189893 0loss/FBt3/assignment_loss‚0.5344180014323053GvÄaÉÛ 2loss/SRqqt1/assignment_loss‚1.7235411230409354 2loss/SRqqt3/assignment_loss‚1.4536480946426695 1loss/FBt1/assignment_loss‚0.23215110733639854 1loss/SRqqt4/detection_loss‚1.0391967262527564 .loss/FBt1/detection_loss‚2.240081305305163 1loss/FBt2/assignment_loss‚0.24876998524332908 1loss/SRqqt3/detection_loss‚2.0255441881240683 .loss/FRt3/detection_loss‚2.320101986080408 1loss/FBt4/assignment_loss‚0.12206977595022242 1loss/SRqqt1/detection_loss‚1.7595456875207132 1loss/SRqqt2/detection_loss‚2.4653382321883894 epoch‚5 _runtime‚5105.560301171 _step‚526 &loss/total_loss‚45.549494017483894 trainer/global_step‚6399 0loss/FRt1/assignment_loss‚3.3503850975940965 0loss/FRt3/assignment_loss‚2.9819231420755385 0loss/FRt4/assignment_loss‚1.9455816074023171 1loss/SRqqt2/assignment_loss‚2.188884316333409 -loss/FRt2/detection_loss‚2.97833835462044 /loss/FBt4/detection_loss‚1.3221336309798062 % _timestamp‚1.7884004809235444e+09 2loss/SRqqt4/assignment_loss‚0.9322130823189893 0loss/FBt3/assignment_loss‚0.5344180014323053 /loss/FRt1/detection_loss‚2.0135522155926147 .loss/FBt3/detection_loss‚2.243388129906221 0loss/FRt2/assignment_loss‚4.3427169487394135 .loss/FBt2/detection_loss‚3.405954910176141 /loss/FRt4/detection_loss‚1.6800163686275482Ž ù›Ê‚(: ê¦ãÔÝÕ€° gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚114.6' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.919999999999998 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\pyMxË‚(:í ê¦ã԰ʲ cpu‚25.36530772503492 disk.sda.out‚ 7625.6875 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2693.87109375 network.sent‚ 1.5417821e+07 network.recv‚ 1.844638e+06" disk.nvme0c0n1.in‚ 378.94140625& disk.nvme0n1.out‚28237.58642578125 disk.drbd2266.out‚0 disk.dm-0.in‚ 378.94140625( proc.memory.availableMB‚ 896.73828125) disk./.usagePercent‚15.40148750208083 disk.sda.in‚ 0.0546875 disk.dm-1.out‚ 7625.6875 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.sda3.in‚ 0.0546875 disk.rbd1.out‚ 14.3779296875 disk.sda2.out‚0 disk.drbd2265.out‚0$ memory_percent‚94.52674388885498* proc.memory.percent‚16.442084312438965% disk./.usageGB‚229.49329376220703# disk.dm-0.out‚28237.58642578125 disk.drbd1716.out‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.nvme0n1.in‚ 378.94140625 disk.dm-1.in‚ 0.0546875( disk.nvme0c0n1.out‚28237.58642578125 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 7625.68756ÂhÌ‚(:’ ù¦ãÔÃÐЮ gpu.0.gpu‚34.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚138.866' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.773200000000003 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ҵ͂(:ö ù¦ãԃщ± proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda.out‚ 7626.1796875% disk./.usageGB‚229.49329376220703 disk.drbd2266.out‚0 disk.nvme0n1.in‚ 378.94140625 disk.sda.in‚ 0.0546875 disk.dm-1.out‚ 7626.1796875 disk.rbd0.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 378.94140625( proc.memory.availableMB‚ 897.23046875 network.sent‚ 1.5526434e+07 network.recv‚ 1.850865e+06 disk.sda1.out‚0 disk.sda3.in‚ 0.0546875 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 378.94140625( disk.nvme0c0n1.out‚28237.78173828125 disk.drbd1716.out‚0# memory_percent‚94.5237398147583 disk.drbd2266.in‚0 disk.drbd2265.in‚0# proc.memory.rssMB‚ 2693.87109375 disk.sda3.out‚ 7626.1796875 disk.rbd1.out‚ 14.4638671875& disk.nvme0n1.out‚28237.78173828125 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.0546875 disk.drbd2265.out‚0# disk.dm-0.out‚28237.78173828125) disk./.usagePercent‚15.40148750208083 disk.sda2.out‚0 disk.sda1.in‚0* proc.memory.percent‚16.442084312438965 cpu‚25.364303720307845ï{¨2Î- _wandbruntime‚5140  _runtime‚5140Ѓõ¹ÈÏ _runtime‚5140.239614493 _step‚527 ' lr-AdamW/pg1‚0.00011371250323441986 ' lr-AdamW/pg2‚0.00011371250323441986 trainer/global_step‚6449 $ _timestamp‚1.788400515602738e+09{rÙrÍÐÇ ' lr-AdamW/pg1‚0.00011371250323441986 ' lr-AdamW/pg2‚0.00011371250323441986 trainer/global_step‚6449 $ _timestamp‚1.788400515602738e+09 _runtime‚5140.239614493 _step‚527w 4‹2Ñ- _wandbruntime‚5140  _runtime‚5140ü,QÚJÒÙO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚502 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9070876728942 trainer/global_step‚6449 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9097097604608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚201929432374272 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚190939509880320 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85033595 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚182053087003890 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚172982375396352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884005156033976e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36661705 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130391252 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86299061 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36661205 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _step‚528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚190939696883200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9097097604608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85033597 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚181842599278592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LDDeviceÚB[\•StatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚190939509880320 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚172982392435712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚181842599278592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9070711607538 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚192674852238336 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 _runtime‚5140.240440482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9096928604672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226136064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85033597 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49637356 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚500 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130411290 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9254580135936 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202195968 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130391252 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9254377939968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 215424847 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚181842581275648 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86298559 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚201929206238208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9096928604672 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49637354 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85033595 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 215444887 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130411290 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚192674828298240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969øüÁä'ÓÞO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9070711607538 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚192674852238336 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 _runtime‚5140.240440482 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9096928604672 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226136064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85033597 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚49637356 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚500 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 130411290 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9254580135936 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202195968 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 130391252 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9254377939968 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 215424847 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚181842581275648 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86298559 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚201929206238208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9096928604672 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚49637354 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85033595 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 215444887 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 130411290 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚192674828298240 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚502 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9070876728942 trainer/global_step‚6449 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9097097604608 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚201929432374272 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚190939509880320 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85033595 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚182053087003890 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚172982375396352 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884005156033976e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚36661705 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 130391252 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86299061 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚36661205 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _step‚528 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚190939696883200 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9097097604608 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85033597 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚181842599278592 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚190939509880320 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚172982392435712 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚181842599278592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199ë’ŸT2Ô- _wandbruntime‚5140  _runtime‚5140)ƒbJé&ÕãM UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9098321742848 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚182080601511742 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 _step‚529 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85046254 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36666745 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚181869850640384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49644731 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 215457293 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚482 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚192703765186560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚181869868643328 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130431103 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚201959540528128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚173008318578688 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9098490756096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚190968172383232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130411039 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 % _timestamp‚1.7884005162637045e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9072265893694 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚181869868643328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚6449 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86311960 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 215477359 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215477359 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36667227 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5140.901070992 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85046256 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192703741246464 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49644733 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9072100772082 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130431103 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9255799281664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚182080419350770 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚190968359399424 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚181869850640384 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9098490756096 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9256003561472 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86311476 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9098321742848 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612ÚGæIÖèM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 130431103 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9255799281664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚182080419350770 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚190968359399424 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚181869850640384 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9098490756096 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9256003561472 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86311476 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9098321742848 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9098321742848 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚182080601511742 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 _step‚529 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85046254 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36666745 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚181869850640384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚49644731 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 215457293 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚482 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚192703765186560 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚181869868643328 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 130431103 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚201959540528128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚173008318578688 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9098490756096 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚190968172383232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 130411039 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 % _timestamp‚1.7884005162637045e+09 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9072265893694 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 A9D; .µ¥ eviceStatsMonitor.on_train_batch_end/segment.all.current‚416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚181869868643328 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 trainer/global_step‚6449 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86311960 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 215477359 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 215477359 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚36667227 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5140.901070992 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85046256 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚192703741246464 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚49644733 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9072100772082 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248‘Ÿå2×- _wandbruntime‚5140  _runtime‚5140¬±êVYØÓ /loss/FRt4/detection_loss‚1.8326722960522834 0loss/FRt3/assignment_loss‚3.0507623297827586 2loss/SRqqt2/assignment_loss‚2.1913575371727347 0loss/FBt1/assignment_loss‚0.8904569091021337 0loss/SRqqt3/detection_loss‚1.880076887872484 -loss/FBt3/detection_loss‚2.75772470777685 /loss/FRt4/assignment_loss‚1.879640666728324 /loss/FRt2/detection_loss‚3.2785097801167034 /loss/FBt2/detection_loss‚2.9796304967668323 /loss/FBt4/detection_loss‚0.7130107011237159 /loss/FRt2/assignment_loss‚4.438534207966017 0loss/FBt2/assignment_loss‚0.5521046654838655 _step‚530 2loss/SRqqt1/assignment_loss‚1.3803620065393096 2loss/SRqqt3/assignment_loss‚1.7488985003696547 0loss/SRqqt4/detection_loss‚0.785777504227179 2loss/SRqqt4/assignment_loss‚1.0943267861764658 1loss/FBt4/assignment_loss‚0.28416926801113174 /loss/FRt3/detection_loss‚2.4230077333596287 0loss/SRqqt1/detection_loss‚2.004569463770498 /loss/FBt1/detection_loss‚2.2660612843253394 %loss/total_loss‚46.24645211116432 $ _timestamp‚1.788400516265564e+09 _runtime‚5140.902307596 1loss/FBt3/assignment_loss‚0.19100684965749018 epoch‚5 trainer/global_step‚6449 0loss/FRt1/assignment_loss‚3.0700708548227946 /loss/FRt1/detection_loss‚2.0791941848066195 1loss/SRqqt2/detection_loss‚2.4745264891535044å§-©^ÙØ 2loss/SRqqt4/assignment_loss‚1.0943267861764658 1loss/FBt4/assignment_loss‚0.28416926801113174 /loss/FRt3/detection_loss‚2.4230077333596287 0loss/SRqqt1/detection_loss‚2.004569463770498 /loss/FBt1/detection_loss‚2.2660612843253394 %loss/total_loss‚46.24645211116432 $ _timestamp‚1.788400516265564e+09 _runtime‚5140.902307596 1loss/FBt3/assignment_loss‚0.19100684965749018 epoch‚5 trainer/global_step‚6449 0loss/FRt1/assignment_loss‚3.0700708548227946 /loss/FRt1/detection_loss‚2.0791941848066195 1loss/SRqqt2/detection_loss‚2.4745264891535044 /loss/FRt4/detection_loss‚1.8326722960522834 0loss/FRt3/assignment_loss‚3.0507623297827586 2loss/SRqqt2/assignment_loss‚2.1913575371727347 0loss/FBt1/assignment_loss‚0.8904569091021337 0loss/SRqqt3/detection_loss‚1.880076887872484 -loss/FBt3/detection_loss‚2.75772470777685 /loss/FRt4/assignment_loss‚1.879640666728324 /loss/FRt2/detection_loss‚3.2785097801167034 /loss/FBt2/detection_loss‚2.9796304967668323 /loss/FBt4/detection_loss‚0.7130107011237159 /loss/FRt2/assignment_loss‚4.438534207966017 0loss/FBt2/assignment_loss‚0.5521046654838655 _step‚530 2loss/SRqqt1/assignment_loss‚1.3803620065393096 2loss/SRqqt3/assignment_loss‚1.7488985003696547 0loss/SRqqt4/detection_loss‚0.785777504227179’§R ÑÚ‚(:† ˆ§ãÔºÒ°® gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚145.595' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.119 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z4Ä¥…Û‚(:ú ˆ§ãԈ䘱* disk./.usagePercent‚15.401487758088365$ disk./.usageGB‚229.4932975769043 disk.drbd2080.out‚0 disk.nvme0n1.in‚ 378.94140625# proc.memory.rssMB‚ 2693.87109375 cpu‚25.363342310959972 disk.sda2.out‚0 disk.dm-1.in‚ 0.0546875 disk.sda3.in‚ 0.0546875( proc.memory.availableMB‚ 896.97265625( disk.nvme0c0n1.out‚28239.00537109375 network.sent‚ 1.5532483e+07 network.recv‚ 1.855302e+06 disk.dm-0.in‚ 378.94140625 disk.sda.in‚ 0.0546875 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 378.94140625 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.sda1.in‚0$ memory_percent‚94.52531337738037 disk.drbd2265.in‚0 disk.drbd2265.out‚0& disk.nvme0n1.out‚28239.00537109375* proc.memory.percent‚16.442084312438965 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.dm-1.out‚ 7626.91796875 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.rbd1.out‚ 14.5263671875 disk.sda.out‚ 7626.91796875 disk.drbd1716.in‚0 disk.drbd2080.in‚0# disk.dm-0.out‚28239.00537109375 disk.sda3.out‚ 7626.91796875({Y]‘Ü‚(:† —§ãÔÕñ€® gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚124.865' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.973 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á½Ý‚(:ö —§ãÔˆÓ·° disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.sda3.in‚ 0.0546875 disk.sda3.out‚ 7627.3203125' proc.memory.availableMB‚ 896.9296875 disk.sda.in‚ 0.0546875 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 14.5263671875$ memory_percent‚94.52557563781738 network.recv‚ 1.862143e+06 disk.dm-1.in‚ 0.0546875 disk.dm-0.in‚ 378.94140625 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 378.94140625& disk.nvme0n1.out‚28239.24365234375% disk./.usageGB‚229.49335479736328 disk.drbd1716.in‚0 disk.dm-1.out‚ 7627.3203125 disk.drbd2265.in‚0 disk.drbd2265.out‚0# disk.dm-0.out‚28239.24365234375# proc.memory.rssMB‚ 2693.87109375 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 378.94140625* proc.memory.percent‚16.442084312438965* disk./.usagePercent‚15.401491598201378 disk.sda.out‚ 7627.3203125 disk.rbd0.in‚0( disk.nvme0c0n1.out‚28239.24365234375 cpu‚25.36244349222609 network.sent‚ 1.5641621e+07 disk.sda2.in‚0ý4í(Þ‚(:’ ¦§ãÔãØ ® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚138.866' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.773200000000003 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0xß‚(:í ¦§ãÔ„…ì± disk.sda.in‚ 0.0546875 disk.sda1.in‚0 disk.sda2.in‚0$ memory_percent‚94.52550411224365( proc.memory.availableMB‚ 896.94140625 disk.drbd2265.out‚0% disk./.usageGB‚229.49328994750977 disk.dm-1.in‚ 0.0546875 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.dm-0.in‚ 378.94140625 disk.sda1.out‚0 disk.rbd1.out‚ 14.55859375 disk.nvme0n1.in‚ 378.94140625 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0" disk.nvme0n1.out‚ 28239.7578125 cpu‚25.361527893787997 proc.cpu.threads‚17 network.recv‚ 1.865363e+06* disk./.usagePercent‚15.401487246073295 disk.rbd0.out‚0 disk.dm-0.out‚ 28239.7578125$ disk.nvme0c0n1.out‚ 28239.7578125 network.sent‚ 1.5647148e+07 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.sda3.out‚ 7627.88671875 disk.sda.out‚ 7627.88671875# proc.memory.rssMB‚ 2693.87109375 disk.dm-1.out‚ 7627.88671875 disk.sda3.in‚ 0.0546875" disk.nvme0c0n1.in‚ 378.94140625 disk.sda2.out‚0* proc.memory.percent‚16.442084312438965³}2à- _wandbruntime‚5175  _runtime‚5175ˆ9e\ÆáÀ _runtime‚ 5175.75129112 _step‚531 & lr-AdamW/pg1‚0.0001121627200406376 & lr-AdamW/pg2‚0.0001121627200406376 trainer/global_step‚6499 % _timestamp‚1.7884005511144083e+09¹$ËâÅ _step‚531 & lr-AdamW/pg1‚0.0001121627200406376 & lr-AdamW/pg2‚0.0001121627200406376 trainer/global_step‚6499 % _timestamp‚1.7884005511144083e+09 _runtime‚ 5175.75129112“fÄ—72ã- _wandbruntime‚5175  _runtime‚5175i®wíÞ'äØO E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131403992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9166599927296 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚6499 _runtime‚5175.752176501 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚183419900099862 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85666545 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131403992 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚183419717939098 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86935815 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194120475709440 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9325273336320 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85666547 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9166768927232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85666545 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 217070539 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚174279551552512 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚174279534513152 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚192372649439744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9166599927296 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚183206067515392 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86936337 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50006206 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50006204 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9325488115200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194120499649536 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 217050499 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85666547 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9140183425946 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9140348547350 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚183206067515392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚183206049512448v Nã'åÝO ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9140183425946 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9140348547350 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚183206067515392 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚183206049512448 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 131403992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9166599927296 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚520 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚6499 _runtime‚5175.752176501 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚183419900099862 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚85666545 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 131403992 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚183419717939098 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚86935815 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚194120475709440 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9325273336320 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚85666547 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9166768927232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚85666545 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 217070539 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚174279551552512 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚174279534513152 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚192372649439744 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9166599927296 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚183206067515392 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚86936337 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50006206 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50006204 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9325488115200 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚194120499649536 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 217050499 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚85666547 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064”ŠAœÞ2æ- _wandbruntime‚5176  _runtime‚5176êÙ©ÏçãM K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131403741 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDe銣š$viceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚183447232446950 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9141572590490 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9168162078720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚183233318877184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚192401498958848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217103011 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36934873 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86948981 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9326690664448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚533 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9167993065472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚192401498958848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131423805 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85679206 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚194149412597760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85679204 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚183233318877184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194149388657664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86948454 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884005517923856e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50013581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚183447050285978 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚183233336880128 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚203476315930624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚174305477695488 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9167993065472 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚174305494734848 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6499 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85679204 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9326903332864 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _runtime‚5176.429558961 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85679206 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9168162078720 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131423805 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50013583 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131403741 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚203476079322112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0º ¿uî&èèM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚194149388657664 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚86948454 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884005517923856e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚50013581 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚183447050285978 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚183233336880128 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚203476315930624 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚174305477695488 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9167993065472 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚525 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚174305494734848 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚6499 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚85679204 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9326903332864 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _runtime‚5176.429558961 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚85679206 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9168162078720 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 131423805 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚527 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50013583 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 131403741 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚203476079322112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 131403741 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚183447232446950 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9141572590490 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9168162078720 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚183233318877184 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚192401498958848 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 217103011 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚36934873 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚86948981 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9326690664448 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚533 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9167993065472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚192401498958848 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 131423805 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚85679206 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚194149412597760 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚85679204 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚183233318877184 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760•oíe2é- _wandbruntime‚5176  _runtime‚5176âĪ%[êÕ /loss/FRt2/detection_loss‚2.8412035597222194 1loss/SRqqt2/detection_loss‚2.4186308957912304 _runtime‚5176.430622408 1loss/SRqqt1/detection_loss‚1.6785363464378844 epoch‚5 0loss/FBt3/assignment_loss‚0.3144570142030706 /loss/FRt1/detection_loss‚2.0660630577140386 2loss/SRqqt3/assignment_loss‚1.4450866575539112 2loss/SRqqt4/assignment_loss‚0.8671631822636461 1loss/SRqqt3/detection_loss‚1.8174158552289008 .loss/FBt2/detection_loss‚3.956852447986603 0loss/FRt2/assignment_loss‚3.8790832800524577 .loss/FBt1/detection_loss‚2.642839769522349 /loss/FBt4/detection_loss‚0.8920396654478585 _step‚534 /loss/FRt4/detection_loss‚1.8972680568695068 %loss/total_loss‚45.25074114684364 0loss/FBt1/assignment_loss‚0.3490507094506433 1loss/FBt2/assignment_loss‚0.48896073102873744 /loss/FRt1/assignment_loss‚3.053806953959995 2loss/SRqqt1/assignment_loss‚1.2353363129876407 1loss/SRqqt2/assignment_loss‚2.152323107338614 1loss/SRqqt4/detection_loss‚1.2595846924711676 1loss/FBt4/assignment_loss‚0.20161770612089766 0loss/FRt4/assignment_loss‚1.6655990967945176 .loss/FRt3/detection_loss‚2.522859051823616 .loss/FBt3/detection_loss‚2.844218305179051 trainer/global_step‚6499 % _timestamp‚1.7884005517938943e+09 0loss/FRt3/assignment_loss‚2.7607446908950806 é`ëÚ _runtime‚5176.430622408 1loss/SRqqt1/detection_loss‚1.6785363464378844 epoch‚5 0loss/FBt3/assignment_loss‚0.3144570142030706 /loss/FRt1/detection_loss‚2.0660630577140386 2loss/SRqqt3/assignment_loss‚1.4450866575539112 2loss/SRqqt4/assignment_loss‚0.8671631822636461 1loss/SRqqt3/detection_loss‚1.8174158552289008 .loss/FBt2/detection_loss‚3.956852447986603 0loss/FRt2/assignment_loss‚3.8790832800524577 .loss/FBt1/detection_loss‚2.642839769522349 /loss/FBt4/detection_loss‚0.8920396654478585 _step‚534 /loss/FRt4/detection_loss‚1.8972680568695068 %loss/total_loss‚45.25074114684364 0loss/FBt1/assignment_loss‚0.3490507094506433 1loss/FBt2/assignment_loss‚0.48896073102873744 /loss/FRt1/assignment_loss‚3.053806953959995 2loss/SRqqt1/assignment_loss‚1.2353363129876407 1loss/SRqqt2/assignment_loss‚2.152323107338614 1loss/SRqqt4/detection_loss‚1.2595846924711676 1loss/FBt4/assignment_loss‚0.20161770612089766 0loss/FRt4/assignment_loss‚1.6655990967945176 .loss/FRt3/detection_loss‚2.522859051823616 .loss/FBt3/detection_loss‚2.844218305179051 trainer/global_step‚6499 % _timestamp‚1.7884005517938943e+09 0loss/FRt3/assignment_loss‚2.7607446908950806 /loss/FRt2/detection_loss‚2.8412035597222194 1loss/SRqqt2/detection_loss‚2.4186308957912304–&"Q7’ì‚(:‡ µ§ãÔôŸ³° gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚128.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.7284 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0@A'Ûví‚(:ë µ§ãÔèÕ»² disk.dm-1.out‚ 7628.34765625# disk.nvme0n1.out‚28239.96484375 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.dm-0.out‚28239.96484375 disk.rbd1.out‚ 14.55859375" disk.nvme0c0n1.in‚ 378.94140625% disk.nvme0c0n1.out‚28239.96484375 disk.drbd2265.in‚0 disk.sda3.out‚ 7628.34765625% proc.memory.availableMB‚ 896.90625 cpu‚25.36062556423074 network.sent‚ 1.565317e+07 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda.in‚ 0.0546875* disk./.usagePercent‚15.401487246073295 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.dm-0.in‚ 378.94140625 disk.sda3.in‚ 0.0546875 disk.rbd1.in‚0 disk.sda.out‚ 7628.34765625* proc.memory.percent‚16.442084312438965 disk.dm-1.in‚ 0.0546875 disk.drbd2265.out‚0 network.recv‚ 1.869886e+06% disk./.usageGB‚229.49328994750977 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 378.94140625 disk.drbd2266.out‚0$ memory_percent‚94.52571868896484# proc.memory.rssMB‚ 2693.87109375@Ü“¡’î‚(:‡ ħãÔ¹“ö­ gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚138.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.6658 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ðç~û„ï‚(:ù ħãÔÇø° disk.nvme0n1.in‚ 378.94140625" disk.nvme0c0n1.in‚ 378.94140625& disk.nvme0n1.out‚28240.60205078125( proc.memory.availableMB‚ 897.02734375 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.drbd1716.in‚0# proc.memory.rssMB‚ 2693.87109375 cpu‚25.359702351551412 proc.cpu.threads‚17 network.sent‚ 1.5761649e+07 network.recv‚ 1.875676e+06 disk.drbd2265.out‚0# disk.dm-0.out‚28240.60205078125 disk.sda1.out‚0 disk.rbd1.out‚14.62158203125 disk.dm-1.in‚ 0.0546875 disk.drbd2266.out‚0 disk.sda.in‚ 0.0546875 disk.sda2.in‚0* disk./.usagePercent‚15.401500046450007% disk./.usageGB‚229.49348068237305( disk.nvme0c0n1.out‚28240.60205078125 disk.dm-1.out‚ 7628.8828125 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.sda3.in‚ 0.0546875 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.sda.out‚ 7628.8828125$ memory_percent‚94.52497959136963 disk.sda1.in‚0* proc.memory.percent‚16.442084312438965 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.dm-0.in‚ 378.94140625 disk.sda3.out‚ 7628.8828125 disk.rbd1.in‚0vñh72ð- _wandbruntime‚5211  _runtime‚5211n&sÉñà ' lr-AdamW/pg1‚0.00011061196374813886 ' lr-AdamW/pg2‚0.00011061196374813886 trainer/global_step‚6549 % _timestamp‚1.7884005869228864e+09 _runtime‚5211.559737405 _step‚535¤g ÒÎòÈ ' lr-AdamW/pg2‚0.00011061196374813886 trainer/global_step‚6549 % _timestamp‚1.7884005869228864e+09 _runtime‚5211.559737405 _step‚535 ' lr-AdamW/pg1‚0.00011061196374813886—†ñ}2ó- _wandbruntime‚5211  _runtime‚5211“©)yôØO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚193805976002048 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 5211.56061423 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9236440249856 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86299497 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9209655244354 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚193805788999168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132396694 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚175576710669312 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚175576693629952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87577196 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86299497 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184569517749248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚184569535752192 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218696191 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195566147060736 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚195566123120640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚193805976002048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9396150727168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86299495 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9236271249920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184569517749248 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86299495 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87577712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204962504178176 _step‚536 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37202142 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193805788999168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9236440249856 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37202656 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9236271249920 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230330368 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚184786348874306 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 % _timestamp‚1.7884005869235668e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚516 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50375054 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132376656 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204962273847808 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132396694 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 132376656 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚514 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9236440249856 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86299497 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9209655244354 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚193805788999168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 132396694 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚175576710669312 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚175576693629952 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚87577196 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86299497 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚184569517749248 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚184569535752192 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 218696191 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚195566147060736 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚195566123120640 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚193805976002048 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9396150727168 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86299495 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9236271249920 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚184569517749248 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚86299495 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚87577712 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚204962504178176 _step‚536 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37202142 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚193805788999168 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9236440249856 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37202656 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9236271249920 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230330368 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚184786348874306 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 % _timestamp‚1.7884005869235668e+09 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚516 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50375054 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 132376656 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚204962273847808 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 132396694˜îú”2ö- _wandbruntime‚5212  _runtime‚5212qo³é&÷ãM OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 132396443 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50382433 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9211209530510 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚5212.231994416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86312156 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87590429 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚175602653851648 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132396443 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204992604664832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _step‚537 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218728663 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚195595036068864 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚193834638518272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37207996 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚195595060008960 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204992837079040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132416507 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7884005875943613e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193834451502080 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9237664388096 trainer/global_step‚6549 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚184813681221186 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87589927 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚193834638518272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚184596787113984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚175602636812288 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218708597 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚500 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9397568595968 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86312154 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 132416507 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218708597 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37207496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9211044408898 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218728663 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 218728663 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚195595036068864 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚193834638518272 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚37207996 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚195595060008960 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚204992837079040 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 132416507 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 % _timestamp‚1.7884005875943613e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚193834451502080 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9237664388096 trainer/global_step‚6549 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚184813681221186 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚87589927 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚193834638518272 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚184596787113984 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚175602636812288 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 218708597 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚500 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9397568595968 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86312154 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 132416507 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 218708597 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚37207496 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9211044408898 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 218728663 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50382433 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9211209530510 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _runtime‚5212.231994416 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86312156 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚87590429 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚175602653851648 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 132396443 [MDeviceStatsMonitor.on_train_bZ¼3]atch_end/inactive_split_bytes.small_pool.current‚ 208474112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚204992604664832 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _step‚537™ ¯w/2ù- _wandbruntime‚5212  _runtime‚5212®A‹"XúÒ 2loss/SRqqt3/assignment_loss‚1.4925936611894615 .loss/FRt2/detection_loss‚3.342021311322848 1loss/SRqqt2/detection_loss‚2.3980896739583266 &loss/total_loss‚47.014491533181456 0loss/FRt4/assignment_loss‚1.9144918303335867 2loss/SRqqt2/assignment_loss‚1.6386758566981083 .loss/FRt3/detection_loss‚2.519284355640411 .loss/FRt4/detection_loss‚1.692310519155956 .loss/FBt1/detection_loss‚2.589716228571805 _runtime‚ 5212.23301931 2loss/SRqqt4/assignment_loss‚1.2804793752046923 1loss/SRqqt3/detection_loss‚2.0138520842943435 /loss/FBt3/detection_loss‚2.9620361924171448 _step‚538 0loss/FBt1/assignment_loss‚0.6156569245185944 1loss/FBt2/assignment_loss‚0.29681450577875995 .loss/FRt1/detection_loss‚1.941292407742793 0loss/SRqqt1/detection_loss‚2.314941171379316 epoch‚5 0loss/FRt2/assignment_loss‚3.7820188055435815 0loss/FRt1/assignment_loss‚3.7828632844121834 0loss/FRt3/assignment_loss‚3.4473612695932387 0loss/FBt4/assignment_loss‚0.2151323444539704 .loss/FBt2/detection_loss‚3.185086836417516 1loss/SRqqt4/detection_loss‚1.0639442720015844 /loss/FBt4/detection_loss‚0.6184806805071225 trainer/global_step‚6549 1loss/SRqqt1/assignment_loss‚1.383822090081161 0loss/FBt3/assignment_loss‚0.5235258519649506 % _timestamp‚1.7884005875962737e+09F8õ]û× 0loss/FBt4/assignment_loss‚0.2151323444539704 .loss/FBt2/detection_loss‚3.185086836417516 1loss/SRqqt4/detection_loss‚1.0639442720015844 /loss/FBt4/detection_loss‚0.6184806805071225 trainer/global_step‚6549 1loss/SRqqt1/assignment_loss‚1.383822090081161 0loss/FBt3/assignment_loss‚0.5235258519649506 % _timestamp‚1.7884005875962737e+09 2loss/SRqqt3/assignment_loss‚1.4925936611894615 .loss/FRt2/detection_loss‚3.342021311322848 1loss/SRqqt2/detection_loss‚2.3980896739583266 &loss/total_loss‚47.014491533181456 0loss/FRt4/assignment_loss‚1.9144918303335867 2loss/SRqqt2/assignment_loss‚1.6386758566981083 .loss/FRt3/detection_loss‚2.519284355640411 .loss/FRt4/detection_loss‚1.692310519155956 .loss/FBt1/detection_loss‚2.589716228571805 _runtime‚ 5212.23301931 2loss/SRqqt4/assignment_loss‚1.2804793752046923 1loss/SRqqt3/detection_loss‚2.0138520842943435 /loss/FBt3/detection_loss‚2.9620361924171448 _step‚538 0loss/FBt1/assignment_loss‚0.6156569245185944 1loss/FBt2/assignment_loss‚0.29681450577875995 .loss/FRt1/detection_loss‚1.941292407742793 0loss/SRqqt1/detection_loss‚2.314941171379316 epoch‚5 0loss/FRt2/assignment_loss‚3.7820188055435815 0loss/FRt1/assignment_loss‚3.7828632844121834 0loss/FRt3/assignment_loss‚3.4473612695932387šãLQ‚ü‚(:’ Ó§ãÔˆ´™® gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚142.932' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.586399999999994 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0„ÃZ€ý‚(:õ Ó§ãÔ…«¶±# disk.dm-0.out‚28240.84423828125 disk.sda3.out‚ 7629.2421875 disk.nvme0n1.in‚ 378.94140625 disk.drbd1716.out‚0 cpu‚25.358811653225757 disk.dm-1.in‚ 0.0546875 disk.drbd2080.in‚0# proc.memory.rssMB‚ 2693.87109375* proc.memory.percent‚16.442084312438965 network.sent‚ 1.576788e+07 disk.dm-0.in‚ 378.94140625& disk.nvme0n1.out‚28240.84423828125 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.sda3.in‚ 0.0546875 disk.rbd1.out‚14.62158203125" disk.nvme0c0n1.in‚ 378.94140625 disk.sda.in‚ 0.0546875 disk.drbd1716.in‚0 disk.drbd2265.out‚0 network.recv‚ 1.88007e+06$ disk./.usageGB‚229.4934539794922 disk.sda1.out‚0 disk.rbd1.in‚0 disk.dm-1.out‚ 7629.2421875$ memory_percent‚94.52052116394043 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0' proc.memory.availableMB‚ 897.7578125 proc.cpu.threads‚17( disk.nvme0c0n1.out‚28240.84423828125 disk.sda.out‚ 7629.2421875 disk.drbd2266.in‚0* disk./.usagePercent‚15.401498254397266/)×’þ‚(:‡ â§ãÔöÍ® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚144.287' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.8574 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£jóÊoÿ‚(:ä â§ãÔÍ÷þ° disk.dm-0.in‚ 378.94140625 disk.sda1.out‚0" disk.nvme0n1.out‚ 28241.9921875 disk.sda.in‚ 0.0546875 network.sent‚ 1.5875849e+07 disk.sda3.in‚ 0.0546875 disk.sda3.out‚ 7629.96875" disk.nvme0c0n1.in‚ 378.94140625$ disk.nvme0c0n1.out‚ 28241.9921875 disk.nvme0n1.in‚ 378.94140625 disk.sda2.in‚0 disk.drbd1716.out‚0$ disk./.usageGB‚229.4933967590332 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.drbd2266.in‚0$ memory_percent‚94.52149868011475# proc.memory.rssMB‚ 2693.87109375 proc.cpu.threads‚17 disk.sda1.in‚0 disk.drbd2266.out‚0* disk./.usagePercent‚15.401494414284253 disk.drbd2080.in‚0 disk.dm-0.out‚ 28241.9921875 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.0546875* proc.memory.percent‚16.442084312438965 disk.rbd1.out‚ 14.6845703125 disk.drbd2265.out‚0 disk.sda.out‚ 7629.96875 disk.dm-1.out‚ 7629.96875( proc.memory.availableMB‚ 897.59765625 network.recv‚ 1.88571e+06 disk.drbd2080.out‚0 disk.sda2.out‚0 cpu‚25.357935319929435Ã.Ôu2€- _wandbruntime‚5246  _runtime‚52460#ÚûÉà ' lr-AdamW/pg1‚0.00010906055753221067 ' lr-AdamW/pg2‚0.00010906055753221067 trainer/global_step‚6599 % _timestamp‚1.7884006223270826e+09 _runtime‚5246.963946367 _step‚539yê+ÛÎ‚È ' lr-AdamW/pg1‚0.00010906055753221067 ' lr-AdamW/pg2‚0.00010906055753221067 trainer/global_step‚6599 % _timestamp‚1.7884006223270826e+09 _runtime‚5246.963946367 _step‚539›·YM?2ƒ- _wandbruntime‚5246  _runtime‚5246^téÞ'„ØO UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚195238928558592 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚195239115561472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37474332 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86932447 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚185932985986048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚185933003988992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9305942572544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37473819 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220321843 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197011794471936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9279127062762 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9306111572480 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86932447 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚515 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9467261879296 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197011770531840 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9467049197568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚6599 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚195239115561472 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 133369358 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚195238928558592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚513 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88218238 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚206479056351232 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚5246.964877995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚176873869786112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50743906 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220301803 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86932445 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9306111572480 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 $ _timestamp‚1.788400622327825e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9305942572544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚185932985986048 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88217723 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _step‚540 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚176873852746752 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚507439040#™ã'…ÝO GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9467261879296 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚197011770531840 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9467049197568 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚6599 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚195239115561472 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 133369358 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚195238928558592 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚513 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚88218238 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚206479056351232 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚5246.964877995 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚176873869786112 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚50743906 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 220301803 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚86932445 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9306111572480 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 $ _timestamp‚1.788400622327825e+09 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9305942572544 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚185932985986048 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚88217723 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _step‚540 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚176873852746752 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚50743904 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚195238928558592 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚195239115561472 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚37474332 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚86932447 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚185932985986048 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚185933003988992 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9305942572544 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚37473819 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 220321843 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚197011794471936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9279127062762 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9306111572480 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚86932447 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚515 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160œûˆò2†- _wandbruntime‚5247  _runtime‚5247¨g»@‡âM C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220354315 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86945104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚195267778077696 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9307504723968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9307335710720 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133409209 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚176899812968448 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9468677066240 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚176899795929088 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9468462300672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220354315 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88231057 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚197040707420160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220334249 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133409209 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88230564 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚185960255350784 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚491 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚186180312156394 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197040683480064 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133389145 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚493 S?DeviceStatsMonitor.on_train_batch_en3ìa¨d/allocated_bytes.all.freed‚195267591061504 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚195267778077696 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚185960273353728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚185960255350784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9280516227306 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0–I#›í&ˆçM HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9280681348918 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 % _timestamp‚1.7884006232223856e+09 _runtime‚ 5247.85991079 _step‚541 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚86945104 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9307504723968 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 133389145 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚206509145780736 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚86945106 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9307335710720 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚50751283 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚186180494317366 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚195267778077696 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚185960273353728 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚185960255350784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9280516227306 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 220354315 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚86945104 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚195267778077696 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9307504723968 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9307335710720 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 133409209 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚176899812968448 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9468677066240 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚176899795929088 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9468462300672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 220354315 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚88231057 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚197040707420160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 220334249 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 133409209 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88230564 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚185960255350784 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚491 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚186180312156394 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚197040683480064 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 133389145 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚493 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚195267591061504 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87565395 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚198457441883136 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9348598881170 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚178171028902912 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚178171011863552 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 134362060 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚6649 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚196672068118016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51112756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚5282.614747966 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87565395 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚207995356859904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9375782895104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9537938916864 _step‚544 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134362060 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚187296454222848 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9538143209984 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51112754 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚196672255120896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚196672068118016 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 134382098 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚87565395 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚198457441883136 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9348598881170 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚178171028902912 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚178171011863552 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 134362060 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚6649 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚196672068118016 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51112756 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚5282.614747966 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚87565395 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚207995356859904 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9375782895104 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9537938916864 _step‚544 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 134362060 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚187296454222848 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9538143209984 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51112754 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚196672255120896 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚196672068118016 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 134382098 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚198486354831360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9377007033344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134401911 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51120133 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87578054 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚187323723587584 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 221959901 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚198486330891264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 221959901 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚208025687875584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚187547125252574 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚196700730620928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 221979967 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9377176046592 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51120131 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9539565458432 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 221979967 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 % _timestamp‚1.7884006586465545e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚178196972085248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9377176046592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134381847 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88870205 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187323741590528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚98553,î&šèM K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 134381847 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚88870205 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚187323741590528 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚198486354831360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9377007033344 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 134401911 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51120133 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚87578054 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚187323723587584 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 221959901 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚198486330891264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 221959901 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚208025687875584 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚187547125252574 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚196700730620928 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 221979967 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9377176046592 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51120131 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9539565458432 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 221979967 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 % _timestamp‚1.7884006586465545e+09 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚178196972085248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9377176046592 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160¡…ŽKÆ2›- _wandbruntime‚5283  _runtime‚5283?E)"XœÒ _runtime‚5283.285169277 2loss/FBt3/assignment_loss‚0.053170195523306596 trainer/global_step‚6649 0loss/FRt1/assignment_loss‚3.5533540046215055 2loss/SRqqt2/assignment_loss‚2.1714489365878857 1loss/SRqqt4/assignment_loss‚1.376023952414592 0loss/FBt1/assignment_loss‚0.2584247522583496 /loss/FBt1/detection_loss‚3.3097103323255266 _step‚546 2loss/SRqqt3/assignment_loss‚1.6688990841074935 .loss/FRt1/detection_loss‚2.334346314072609 /loss/FBt4/detection_loss‚0.7875614510241734 1loss/FBt4/assignment_loss‚0.20797106314140149 .loss/FRt2/detection_loss‚3.330215836826124 /loss/FBt2/assignment_loss‚0.208955969203088 1loss/SRqqt2/detection_loss‚2.5474473865408647 1loss/SRqqt3/detection_loss‚1.8004022507735018 .loss/FRt3/assignment_loss‚2.93026129352419 /loss/FBt2/detection_loss‚2.9459689930081367 epoch‚5 /loss/FRt4/assignment_loss‚1.845612814613417 .loss/FRt3/detection_loss‚2.461061163952476 .loss/FRt4/detection_loss‚1.859370256975001 &loss/total_loss‚47.709275194701405 /loss/FRt2/assignment_loss‚3.885071691713835 2loss/SRqqt1/assignment_loss‚1.7402017379694796 1loss/SRqqt1/detection_loss‚1.9037498457634703 1loss/SRqqt4/detection_loss‚0.9746889032920202 .loss/FBt3/detection_loss‚3.555356964468956 % _timestamp‚1.7884006586484194e+09ãi~{d× 1loss/FBt4/assignment_loss‚0.20797106314140149 .loss/FRt2/detection_loss‚3.330215836826124 /loss/FBt2/assignment_loss‚0.208955969203088 1loss/SRqqt2/detection_loss‚2.5474473865408647 1loss/SRqqt3/detection_loss‚1.8004022507735018 .loss/FRt3/assignment_loss‚2.93026129352419 /loss/FBt2/detection_loss‚2.9459689930081367 epoch‚5 /loss/FRt4/assignment_loss‚1.845612814613417 .loss/FRt3/detection_loss‚2.461061163952476 .loss/FRt4/detection_loss‚1.859370256975001 &loss/total_loss‚47.709275194701405 /loss/FRt2/assignment_loss‚3.885071691713835 2loss/SRqqt1/assignment_loss‚1.7402017379694796 1loss/SRqqt1/detection_loss‚1.9037498457634703 1loss/SRqqt4/detection_loss‚0.9746889032920202 .loss/FBt3/detection_loss‚3.555356964468956 % _timestamp‚1.7884006586484194e+09 _runtime‚5283.2851692cÉyù77 2loss/FBt3/assignment_loss‚0.053170195523306596 trainer/global_step‚6649 0loss/FRt1/assignment_loss‚3.5533540046215055 2loss/SRqqt2/assignment_loss‚2.1714489365878857 1loss/SRqqt4/assignment_loss‚1.376023952414592 0loss/FBt1/assignment_loss‚0.2584247522583496 /loss/FBt1/detection_loss‚3.3097103323255266 _step‚546 2loss/SRqqt3/assignment_loss‚1.6688990841074935 .loss/FRt1/detection_loss‚2.334346314072609 /loss/FBt4/detection_loss‚0.7875614510241734¢m€ST’ž‚(:‡ ž¨ãÔÎó›® gpu.0.gpu‚11.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚124.576' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.9152 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­ë.„Ÿ‚(:ù ž¨ãÔàìÁ° disk.drbd2266.out‚0' proc.memory.availableMB‚ 897.3828125 proc.cpu.threads‚17* disk./.usagePercent‚15.401796503174658 disk.sda3.in‚ 0.0546875( disk.nvme0c0n1.out‚28249.10791015625 disk.nvme0n1.in‚ 378.94140625 disk.drbd2265.out‚0 disk.drbd2080.out‚0* proc.memory.percent‚16.442084312438965 disk.rbd1.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0# memory_percent‚94.5228099822998 disk.sda1.in‚0 disk.rbd1.out‚ 14.6845703125 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 7631.96484375& disk.nvme0n1.out‚28249.10791015625 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0" disk.nvme0c0n1.in‚ 378.94140625 disk.sda.out‚ 7631.96484375 disk.dm-0.in‚ 378.94140625 network.sent‚ 1.6002277e+07 disk.sda2.in‚0 disk.dm-1.in‚ 0.0546875# disk.dm-0.out‚28249.10791015625 network.recv‚ 1.904495e+06 disk.sda1.out‚0# proc.memory.rssMB‚ 2693.87109375 cpu‚25.354420088045856% disk./.usageGB‚229.49789810180664 disk.sda.in‚ 0.0546875 disk.dm-1.out‚ 7631.96484375Û<’ ‚(:‡ ­¨ãÔÊû ® gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚129.179' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.8358 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚02;ã&„¡‚(:ù ­¨ãÔýñŒ² disk.dm-1.in‚ 0.0546875 disk.dm-0.in‚ 378.94140625 disk.sda2.in‚0* proc.memory.percent‚16.442084312438965 network.recv‚ 1.910713e+06 disk.rbd1.in‚0$ memory_percent‚94.52433586120605 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda3.out‚ 7632.54296875 disk.sda2.out‚0 disk.drbd1716.in‚0' proc.memory.availableMB‚ 897.1328125% disk./.usageGB‚229.49785232543945 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 378.94140625 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda.out‚ 7632.54296875 network.sent‚ 1.6110813e+07 disk.dm-1.out‚ 7632.54296875 disk.drbd2266.out‚0 disk.drbd2265.in‚0" disk.nvme0c0n1.in‚ 378.94140625( disk.nvme0c0n1.out‚28249.33447265625& disk.nvme0n1.out‚28249.33447265625 disk.drbd1716.out‚0* disk./.usagePercent‚15.401793431084245 disk.rbd0.in‚0# disk.dm-0.out‚28249.33447265625 disk.rbd1.out‚ 14.6845703125# proc.memory.rssMB‚ 2693.87109375 cpu‚25.35354852815911 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.0546875 disk.sda.in‚ 0.0546875èÓåB2¢- _wandbruntime‚5318  _runtime‚5318ãïpãɣà _runtime‚5318.331331319 _step‚547 ' lr-AdamW/pg1‚0.00010595708864105186 ' lr-AdamW/pg2‚0.00010595708864105186 trainer/global_step‚6699 % _timestamp‚1.7884006936944802e+09;GžÎ¤È ' lr-AdamW/pg2‚0.00010595708864105186 trainer/global_step‚6699 % _timestamp‚1.7884006936944802e+09 _runtime‚5318.331331319 _step‚547 ' lr-AdamW/pg1‚0.00010595708864105186£X2¥- _wandbruntime‚5318  _runtime‚5318K• ±ß'¦ÙO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88198347 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884006936951427e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209512143270912 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9445454217728 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135354762 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚188886241679930 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚188659940462592 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚188659922459648 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚188659940462592 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚89498095 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135374800 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135374800 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚548 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 223573147 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚199903065354240 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚198105394680320 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88198345 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9445454217728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9445285217792 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135354762 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚188886423840694 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚198105207677440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9418235820982 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88198347 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51481604 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9609053976576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚179468188019712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚179468170980352 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚557 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51481606 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚198105207677440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚555 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 _runtime‚5318.332125587 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26H‡™Oä'§ÞO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚548 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 223573147 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚199903065354240 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚198105394680320 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88198345 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9445454217728 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9445285217792 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 135354762 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚188886423840694 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚198105207677440 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9418235820982 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚88198347 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51481604 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9609053976576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚179468188019712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚179468170980352 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚557 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51481606 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚198105207677440 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚555 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 _runtime‚5318.332125587 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚88198347 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884006936951427e+09 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚209512143270912 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9445454217728 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 135354762 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚188886241679930 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚188659940462592 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚188659922459648 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚188659940462592 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚89498095 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 135374800 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 135374800¤Aí£42¨- _wandbruntime‚5319  _runtime‚5319v4ÒG©ãM @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88211004 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89510809 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209542477877760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚188687209827328 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135374549 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚188687191824384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 223605619 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚198134057196544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51488983 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚199932002242560 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚188913756187782 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚179494114162688 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188913574026810 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _runtime‚5319.001900843 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚6699 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚188687191824384 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9446847369216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9610260869632 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 223605619 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199931978302464 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚X# !Y199 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 _step‚549 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9446678355968 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚198133870180352 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88211004 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚89510809 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚209542477877760 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚188687209827328 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 135374549 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚188687191824384 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 223605619 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚198134057196544 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚51488983 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚199932002242560 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚188913756187782 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚179494114162688 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚188913574026810 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _runtime‚5319.001900843 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 trainer/global_step‚6699 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚188687191824384 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9446847369216 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9610260869632 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 223605619 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚199931978302464 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 _step‚549 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9446678355968¥5š:~2«- _wandbruntime‚5319  _runtime‚5319ˆ`¹V¬Ð 0loss/FRt3/assignment_loss‚2.9531849468908002 0loss/FBt2/assignment_loss‚0.9015634879469824 1loss/FBt4/assignment_loss‚0.26878259866828763 .loss/FRt3/detection_loss‚3.018906299144991 -loss/FRt4/detection_loss‚1.93889330746606 /loss/FBt1/detection_loss‚2.4040200233459474 trainer/global_step‚6699 0loss/FRt4/assignment_loss‚2.0352742274602256 1loss/SRqqt3/detection_loss‚1.8077582196755844 2loss/SRqqt3/assignment_loss‚1.7442615908655252 0loss/FBt3/assignment_loss‚0.7879855278006289 /loss/FRt1/detection_loss‚1.8734692344589838 1loss/SRqqt4/detection_loss‚1.2037580360525422 /loss/FBt4/detection_loss‚0.7052991500037701 /loss/FRt2/assignment_loss‚4.554278285879838 0loss/SRqqt2/detection_loss‚2.636001946597264 epoch‚5 % _timestamp‚1.7884006943661177e+09 /loss/FRt2/detection_loss‚3.4197478231630827 0loss/FRt1/assignment_loss‚3.3683072092041138 1loss/SRqqt1/assignment_loss‚1.562134489486063 2loss/SRqqt2/assignment_loss‚1.9744608854425365 $loss/total_loss‚48.8837913801036 _step‚550 1loss/SRqqt1/detection_loss‚1.6725699818796582 .loss/FBt2/detection_loss‚3.599183589220047 /loss/FBt3/detection_loss‚2.5599541664123535 _runtime‚5319.002848451 0loss/SRqqt4/assignment_loss‚1.30182629698954 0loss/FBt1/assignment_loss‚0.5921700560487807g¥Y[­Õ 0loss/FBt3/assignment_loss‚0.7879855278006289 /loss/FRt1/detection_loss‚1.8734692344589838 1loss/SRqqt4/detection_loss‚1.2037580360525422 /loss/FBt4/detection_loss‚0.7052991500037701 /loss/FRt2/assignment_loss‚4.554278285879838 0loss/SRqqt2/detection_loss‚2.636001946597264 epoch‚5 % _timestamp‚1.7884006943661177e+09 /loss/FRt2/detection_loss‚3.4197478231630827 0loss/FRt1/assignment_loss‚3.3683072092041138 1loss/SRqqt1/assignment_loss‚1.562134489486063 2loss/SRqqt2/assignment_loss‚1.9744608854425365 $loss/total_loss‚48.8837913801036 _step‚550 1loss/SRqqt1/detection_loss‚1.6725699818796582 .loss/FBt2/detection_loss‚3.599183589220047 /loss/FBt3/detection_loss‚2.5599541664123535 _runtime‚5319.002848451 0loss/SRqqt4/assignment_loss‚1.30182629698954 0loss/FBt1/assignment_loss‚0.5921700560487807 0loss/FRt3/assignment_loss‚2.9531849468908002 0loss/FBt2/assignment_loss‚0.9015634879469824 1loss/FBt4/assignment_loss‚0.26878259866828763 .loss/FRt3/detection_loss‚3.018906299144991 -loss/FRt4/detection_loss‚1.93889330746606 /loss/FBt1/detection_loss‚2.4040200233459474 trainer/global_step‚6699 0loss/FRt4/assignment_loss‚2.0352742274602256 1loss/SRqqt3/detection_loss‚1.8077582196755844 2loss/SRqqt3/assignment_loss‚1.7442615908655252¦åi®‚(:’ ¼¨ãÔßäê® gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚142.106' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.421199999999995 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ”žR‚¯‚(:÷ ¼¨ãÔ…좰* disk./.usagePercent‚15.401859225020544# disk.dm-0.out‚28251.00244140625 disk.sda3.out‚ 7632.8984375( proc.memory.availableMB‚ 897.12890625 disk.drbd2080.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 378.94140625 disk.drbd2266.out‚0* proc.memory.percent‚16.442084312438965 disk.sda1.in‚0 network.recv‚ 1.91488e+06 disk.drbd2265.in‚0 disk.sda.in‚ 0.0546875 disk.sda.out‚ 7632.8984375 disk.dm-1.in‚ 0.0546875 disk.dm-1.out‚ 7632.8984375 disk.drbd2266.in‚0% disk./.usageGB‚229.49883270263672& disk.nvme0n1.out‚28251.00244140625 disk.sda2.out‚0$ memory_percent‚94.52435970306396 proc.cpu.threads‚17 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.sda1.out‚0 network.sent‚ 1.6116922e+07 disk.rbd0.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚28251.00244140625# proc.memory.rssMB‚ 2693.87109375 disk.drbd2265.out‚0 disk.dm-0.in‚ 378.94140625 disk.rbd1.out‚ 14.6845703125 cpu‚25.352715440521898 disk.drbd2080.in‚0 disk.sda3.in‚ 0.0546875 disk.nvme0n1.in‚ 378.94140625 disk.drbd1716.in‚0:,±°‚(:’ ˨ãÔ’ž©® gpu.0.gpu‚34.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚139.396' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.879199999999997 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0,y˜¥…±‚(:ú ˨ãÔ‡ü·² disk.sda2.in‚0 disk.drbd1716.in‚0( proc.memory.availableMB‚ 897.12109375( disk.nvme0c0n1.out‚28251.19384765625 disk.drbd2266.out‚0# proc.memory.rssMB‚ 2693.87109375* proc.memory.percent‚16.442084312438965 proc.cpu.threads‚17& disk.nvme0n1.out‚28251.19384765625 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 378.94140625" disk.nvme0c0n1.in‚ 378.94140625 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda3.in‚ 0.0546875 network.sent‚ 1.6224826e+07 network.recv‚ 1.920481e+06$ disk./.usageGB‚229.4988899230957 disk.sda.in‚ 0.0546875 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 14.6845703125* disk./.usagePercent‚15.401863065133558 disk.nvme0n1.in‚ 378.94140625 disk.dm-1.out‚ 7633.26953125 disk.sda3.out‚ 7633.26953125$ memory_percent‚94.52440738677979 cpu‚25.351847811208636 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.0546875# disk.dm-0.out‚28251.19384765625 disk.sda.out‚ 7633.26953125ÚŒo 2²- _wandbruntime‚5353  _runtime‚5353à”‘É³Ã ' lr-AdamW/pg1‚0.00010440567272408656 ' lr-AdamW/pg2‚0.00010440567272408656 trainer/global_step‚6749 % _timestamp‚1.7884007291631975e+09 _runtime‚5353.800042258 _step‚551$PöÎ´È trainer/global_step‚6749 % _timestamp‚1.7884007291631975e+09 _runtime‚5353.800042258 _step‚551 ' lr-AdamW/pg1‚0.00010440567272408656 ' lr-AdamW/pg2‚0.00010440567272408656§!M2µ- _wandbruntime‚5353  _runtime‚5353¹_„ß'¶ÙO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38287874 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 225198799 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9487542517986 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199538347236864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9679702922240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚199538534239744 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚190023390696448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9515125540352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚6749 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9514956540416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90138328 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 % _timestamp‚1.7884007291639574e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9515125540352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚180765347136512 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚199538534239744 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88831295 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199538347236864 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚201348712765440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51850454 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚180765330097152 _runtime‚5353.800981711 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 225178759 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51850456 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚6749 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9514956540416 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90138328 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 % _timestamp‚1.7884007291639574e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9515125540352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚180765347136512 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚199538534239744 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚88831295 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚199538347236864 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚201348712765440 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚51850454 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚180765330097152 _runtime‚5353.800981711 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 225178759 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚51850456 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 225198799 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9487542517986 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚199538347236864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9679702922240 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚199538534239744 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚190023390696448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9515125540352 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992¨“Ši2¸- _wandbruntime‚5354  _runtime‚5354ßtT’é&¹ãM PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _step‚553 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚190050660061184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136387315 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90151407 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚180791273279488 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88843954 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9516349678592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51857831 trainer/global_step‚6749 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9516518691840 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚190050660061184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚180791290318848 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90151918 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚511 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚201377649653760 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚199567009739776 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88843956 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88843954 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225231271 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚211058977654784 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚190280204962018 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9681328001024 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136367251 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 225211205 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38293576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884007298290946e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚199567196755968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 225231271 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136387315 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚199567196755968 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38294085ˆ”:î&ºèM E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚511 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚201377649653760 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚199567009739776 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚88843956 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚88843954 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 225231271 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚211058977654784 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚190280204962018 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9681328001024 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 136367251 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 225211205 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38293576 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884007298290946e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚199567196755968 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 225231271 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 136387315 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚199567196755968 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38294085 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _step‚553 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚190050660061184 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 136387315 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90151407 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚180791273279488 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚88843954 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9516349678592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚51857831 trainer/global_step‚6749 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9516518691840 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚190050660061184 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚180791290318848 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90151918 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0©çýðX2»- _wandbruntime‚5354  _runtime‚5354Õ}x8]¼× 0loss/FRt4/assignment_loss‚1.4692334400282965 .loss/FBt3/detection_loss‚2.528726683722602 %loss/total_loss‚46.15045604864727 1loss/FBt2/assignment_loss‚0.42814500707462294 1loss/FBt4/assignment_loss‚0.19375156118968956 _step‚554 2loss/SRqqt4/assignment_loss‚1.1105166047434263 -loss/FRt3/detection_loss‚2.15604782967191 /loss/FBt2/detection_loss‚2.4862505282674516 epoch‚5 0loss/FRt3/assignment_loss‚2.7896549262498556 2loss/SRqqt2/assignment_loss‚1.9455904715384047 1loss/FBt3/assignment_loss‚0.21534532801403353 /loss/FRt1/detection_loss‚2.0501087935347306 /loss/FRt4/detection_loss‚1.8123831566837099 1loss/SRqqt3/detection_loss‚1.7965385239748728 1loss/SRqqt4/detection_loss‚1.1393322359440448 /loss/FBt4/detection_loss‚0.9210609937888674 /loss/FRt2/assignment_loss‚5.197875278752025 2loss/SRqqt3/assignment_loss‚1.6076617544250829 /loss/FBt1/detection_loss‚3.3779935439427695 _runtime‚5354.467479354 0loss/FBt1/assignment_loss‚0.4149033934360394 1loss/SRqqt1/detection_loss‚1.5594800477131054 trainer/global_step‚6749 2loss/SRqqt1/assignment_loss‚1.8903583003064761 .loss/FRt2/detection_loss‚2.844655999323217 1loss/SRqqt2/detection_loss‚2.7019753009080887 % _timestamp‚1.7884007298307588e+09 0loss/FRt1/assignment_loss‚3.5128663454139444S¥Ûb½Ü 2loss/SRqqt4/assignment_loss‚1.1105166047434263 -loss/FRt3/detection_loss‚2.15604782967191 /loss/FBt2/detection_loss‚2.4862505282674516 epoch‚5 0loss/FRt3/assignment_loss‚2.7896549262498556 2loss/SRqqt2/assignment_loss‚1.9455904715384047 1loss/FBt3/assignment_loss‚0.21534532801403353 /loss/FRt1/detection_loss‚2.0501087935347306 /loss/FRt4/detection_loss‚1.8123831566837099 1loss/SRqqt3/detection_loss‚1.7965385239748728 1loss/SRqqt4/detection_loss‚1.1393322359440448 /loss/FBt4/detection_loss‚0.9210609937888674 /loss/FRt2/assignment_loss‚5.197875278752025 2loss/SRqqt3/assignment_loss‚1.6076617544250829 /loss/FBt1/detection_loss‚3.3779935439427695 _runtime‚5354.467479354 0loss/FBt1/assignment_loss‚0.4149033934360394 1loss/SRqqt1/detection_loss‚1.5594800477131054 trainer/global_step‚6749 2loss/SRqqt1/assignment_loss‚1.8903583003064761 .loss/FRt2/detection_loss‚2.844655999323217 1loss/SRqqt2/detection_loss‚2.7019753009080887 % _timestamp‚1.7884007298307588e+09 0loss/FRt1/assignment_loss‚3.5128663454139444 0loss/FRt4/assignment_loss‚1.4692334400282965 .loss/FBt3/detection_loss‚2.528726683722602 %loss/total_loss‚46.15045604864727 1loss/FBt2/assignment_loss‚0.42814500707462294 1loss/FBt4/assignment_loss‚0.19375156118968956 _step‚554ªÇc»¿œ¾‚(:‘ Ú¨ãÔ澘® gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚125.01' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.002000000000002 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Z˜ y{¿‚(:ð Ú¨ãÔ˽ë°" disk.nvme0c0n1.in‚ 378.94140625 disk.dm-1.in‚ 0.0546875 disk.drbd2266.out‚0 proc.cpu.threads‚17 network.recv‚ 1.923906e+06% disk./.usageGB‚229.49883270263672 disk.nvme0n1.in‚ 378.94140625 disk.sda.out‚ 7633.78125& proc.memory.availableMB‚ 897.140625 disk.drbd2080.out‚0# disk.dm-0.out‚28251.59423828125 disk.sda1.in‚0 disk.sda3.out‚ 7633.78125 disk.rbd1.out‚ 14.6845703125 disk.drbd1716.out‚0 disk.drbd2265.in‚0( disk.nvme0c0n1.out‚28251.59423828125& disk.nvme0n1.out‚28251.59423828125 disk.dm-1.out‚ 7633.78125 disk.drbd2265.out‚0 network.sent‚ 1.6230334e+07 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.sda.in‚ 0.0546875 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.dm-0.in‚ 378.94140625 disk.sda2.in‚0$ memory_percent‚94.52428817749023 cpu‚25.350975374272984 disk.rbd0.out‚0 disk.sda3.in‚ 0.0546875# proc.memory.rssMB‚ 2693.87109375* proc.memory.percent‚16.442084312438965* disk./.usagePercent‚15.401859225020544 disk.rbd0.in‚0 disk.sda1.out‚0þ=µ’À‚(:‡ é¨ãÔ¼˜•® gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚144.528' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.9056 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0)Ât…Á‚(:ú é¨ãÔÖ´ü° disk.drbd2080.in‚0" disk.nvme0c0n1.in‚ 378.94140625 disk.sda.in‚ 0.0546875* proc.memory.percent‚16.442084312438965 disk.drbd2080.out‚0 disk.rbd1.out‚ 14.6845703125 disk.nvme0n1.in‚ 378.94140625 disk.sda2.in‚0 disk.dm-1.out‚ 7634.40234375# memory_percent‚94.5239782333374% disk./.usageGB‚229.49886322021484 disk.sda.out‚ 7634.40234375 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 proc.cpu.threads‚17 network.recv‚ 1.930096e+06 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 7634.40234375 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.dm-0.in‚ 378.94140625 disk.sda3.in‚ 0.0546875( disk.nvme0c0n1.out‚28251.85986328125& disk.nvme0n1.out‚28251.85986328125# proc.memory.rssMB‚ 2693.87109375 network.sent‚ 1.6338796e+07 disk.sda1.out‚0( proc.memory.availableMB‚ 897.19140625 disk.drbd2265.in‚0# disk.dm-0.out‚28251.85986328125 disk.sda1.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.0546875* disk./.usagePercent‚15.401861273080819 disk.drbd2265.out‚0 cpu‚25.350137249467487i>¡’‚(:‡ ø¨ãÔ—½‘® gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚141.577' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.3154 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ílŸYÂ(:ð ø¨ãÔÖÑš±$ disk./.usageGB‚229.4988784790039 disk.nvme0n1.in‚ 378.94140625 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 7635.07421875# proc.memory.rssMB‚ 2693.87109375* proc.memory.percent‚16.442084312438965 disk.dm-0.in‚ 378.94140625& disk.nvme0c0n1.out‚28252.60EÆî€"3515625 disk.sda.in‚ 0.0546875 disk.dm-1.in‚ 0.0546875 cpu‚25.3492566894978 disk.sda.out‚ 7635.07421875 disk.drbd2266.out‚0 network.sent‚ 1.6344848e+07 disk.sda1.out‚0 disk.drbd2266.in‚0$ memory_percent‚94.52352523803711 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0$ disk.nvme0n1.out‚28252.603515625 disk.rbd1.out‚ 14.6845703125 disk.sda2.in‚0& proc.memory.availableMB‚ 897.265625! disk.dm-0.out‚28252.603515625* disk./.usagePercent‚15.401862297110958 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 7635.07421875" disk.nvme0c0n1.in‚ 378.94140625 network.recv‚ 1.934658e+06 disk.sda1.in‚0 disk.sda3.in‚ 0.0546875 disk.rbd1.in‚0 proc.cpu.threads‚17ýÂ?K2Ä- _wandbruntime‚5389  _runtime‚5389î•›ÈÅ _step‚555 ' lr-AdamW/pg1‚0.00010285490026543902 ' lr-AdamW/pg2‚0.00010285490026543902 trainer/global_step‚6799 % _timestamp‚1.7884007646094277e+09 _runtime‚ 5389.24637697é7sªÍÆÇ _runtime‚ 5389.24637697 _step‚555 ' lr-AdamW/pg1‚0.00010285490026543902 ' lr-AdamW/pg2‚0.00010285490026543902 trainer/global_step‚6799 % _timestamp‚1.7884007646094277e+09«‰µ¦2Ç- _wandbruntime‚5389  _runtime‚5389˜2|Þ'ÈØO @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9584627863040 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9584796862976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚511 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 trainer/global_step‚6799 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚202794384116736 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9557014336394 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚200971486796288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52219306 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38559493 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚191619685711110 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 226824451 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9584627863040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚212544974711808 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9750614535168 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89464247 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9584796862976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52219304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9750825119744 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137340166 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚202794360176640 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚182062506253312 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9557179457798 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89464245 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚191386858933248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚191619503550346 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _runtime‚5389.247223107 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚556 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚200971673799168 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90778797 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137360204 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚182062489213952 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚200971486796288 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 226804411 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚191386858933248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90779310 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9584627863040 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚212544974711808 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9750614535168 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚89464247 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9584796862976 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52219304 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9750825119744 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 137340166 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚202794360176640 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚182062506253312 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9557179457798 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚89464245 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚191386858933248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚191619503550346 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _runtime‚5389.247223107 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚556 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚200971673799168 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚90778797 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 137360204 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚182062489213952 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚200971486796288 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 226804411 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚191386858933248 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚90779310 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚200971486796288 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52219306 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚38559493 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚191619685711110¬ Aê2Ê- _wandbruntime‚5389  _runtime‚5389•©óè&ËâM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137359953 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89476904 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89476906 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚182088449435648 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚6799 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52226683 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _runtime‚ 5389.91122842 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚191414146300928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89476904 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚191414128297984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226856923 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9586190014464 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9586021001216 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚191414146300928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226856923 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚191414128297984 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38565314 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90791493 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90791997 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0¼éN—ÄÌçM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9752063890944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚504 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9558568622550 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚38565314 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚90791493 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚90791997 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 137359953 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚89476904 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚89476906 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚182088449435648 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚6799 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚52226683 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _runtime‚ 5389.91122842 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚191414146300928 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚89476904 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚191414128297984 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 226856923 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9586190014464 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9586021001216 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚191414146300928 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 226856923 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚191414128297984 FDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91420835 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90097197 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 228430063 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚204240031527936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚183359665370112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚183359648330752 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 trainer/global_step‚6849 % _timestamp‚1.7884008000012429e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9654468185600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚214061542893056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9654299185664 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚214061781612032 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚192750327170048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚192750327170048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38832679 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚202404626355712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52588154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9821750084096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26¯•ž»ÙÞO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 138352906 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 138332868 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚202404626355712 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚202404813358592 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚192750345172992 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9821535305216 _runtime‚5424.638423434 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚91420835 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90097197 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 228430063 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚204240031527936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚183359665370112 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚183359648330752 UGDeviceStatsMonitor.on_train_batch_start/ac@{ö.W tive_bytes.small_pool.current‚ 168999936 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 trainer/global_step‚6849 % _timestamp‚1.7884008000012429e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9654468185600 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚214061542893056 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9654299185664 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚214061781612032 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚192750327170048 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚192750327170048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚38832679 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚202404626355712 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚52588154 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9821750084096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚192986134485554 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0°ò$šf2Ú- _wandbruntime‚5425  _runtime‚5425’”žé&ÛãM WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138372719 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚561 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38837470 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚204268920536064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138352655 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚202433288858624 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9655861337088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚202433475874816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90109856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚183385608552448 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9627875319346 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138372719 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9822959128576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228482575 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91433001 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52595531 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚204268944476160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 228482575 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9655861337088 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚202433475874816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚214091879664640 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9628040440958 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884008006759214e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚214092105787392 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9823161311232 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚193013648993406 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚183385591513088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90109856 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138352655 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90109854 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9655692323840 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202182656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0ªÇWî&ÜèM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚193013648993406 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚183385591513088 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90109856 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 138352655 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚90109854 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9655692323840 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202182656 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 138372719 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚561 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚38837470 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚204268920536064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 138352655 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚202433288858624 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9655861337088 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚202433475874816 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90109856 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚183385608552448 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9627875319346 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 138372719 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9822959128576 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 228482575 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚91433001 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52595531 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚204268944476160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 228482575 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9655861337088 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚202433475874816 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚214091879664640 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9628040440958 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884008006759214e+09 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚214092105787392 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9823161311232± åð|2Ý- _wandbruntime‚5425  _runtime‚5425£ã ÇXÞÒ /loss/FRt3/assignment_loss‚3.241727650165558 1loss/SRqqt3/assignment_loss‚1.451739428192377 1loss/FBt4/assignment_loss‚0.19414034159298268 0loss/SRqqt1/detection_loss‚1.796470539949157 /loss/FRt3/detection_loss‚2.4586243497770885 /loss/FRt1/detection_loss‚1.9594457128591705 1loss/SRqqt3/detection_loss‚1.7211978484283794 -loss/FBt1/detection_loss‚2.52640694805554 /loss/FBt4/detection_loss‚0.8923616251908243 trainer/global_step‚6849 /loss/FRt2/assignment_loss‚3.416719777243478 0loss/FRt4/assignment_loss‚1.4661324217915535 0loss/FBt2/assignment_loss‚0.4057816397398331 /loss/FRt2/detection_loss‚3.4809168690726873 %loss/total_loss‚45.15725121324702 .loss/FRt1/assignment_loss‚3.14695659122969 2loss/SRqqt2/assignment_loss‚2.6014087136302675 2loss/SRqqt4/assignment_loss‚1.1057992880072058 1loss/FBt3/assignment_loss‚0.01443622992762054 0loss/SRqqt2/detection_loss‚2.544061160513333 1loss/SRqqt4/detection_loss‚1.1677036730829498 /loss/FRt4/detection_loss‚2.0750270899385215 % _timestamp‚1.7884008006776502e+09 _step‚562 2loss/SRqqt1/assignment_loss‚1.7165665850720622 0loss/FBt1/assignment_loss‚0.6660948411693296 /loss/FBt2/detection_loss‚2.5030892425113254 /loss/FBt3/detection_loss‚2.6044426461060843 epoch‚5 _runtime‚5425.3143983059õþ¢]ß× /loss/FRt3/assignment_loss‚3.241727650165558 1loss/SRqqt3/assignment_loss‚1.451739428192377 1loss/FBt4/assignment_loss‚0.19414034159298268 0loss/SRqqt1/detection_loss‚1.796470539949157 /loss/FRt3/detection_loss‚2.4586243497770885 /loss/FRt1/detection_loss‚1.9594457128591705 1loss/SRqqt3/detection_loss‚1.7211978484283794 -loss/FBt1/detection_loss‚2.52640694805554 /loss/FBt4/detection_loss‚0.8923616251908243 trainer/global_step‚6849 /loss/FRt2/assignment_loss‚3.416719777243478 0loss/FRt4/assignment_loss‚1.4661324217915535 0loss/FBt2/assignment_loss‚0.4057816397398331 /loss/FRt2/detection_loss‚3.4809168690726873 %loss/total_loss‚45.15725121324702 .loss/FRt1/assignment_loss‚3.14695659122969 2loss/SRqqt2/assignment_loss‚2.6014087136302675 2loss/SRqqt4/assignment_loss‚1.1057992880072058 1loss/FBt3/assignment_loss‚0.01443622992762054 0loss/SRqqt2/detection_loss‚2.544061160513333 1loss/SRqqt4/detection_loss‚1.1677036730829498 /loss/FRt4/detection_loss‚2.0750270899385215 % _timestamp‚1.7884008006776502e+09 _step‚562 2loss/SRqqt1/assignment_loss‚1.7165665850720622 0loss/FBt1/assignment_loss‚0.6660948411693296 /loss/FBt2/detection_loss‚2.5030892425113254 /loss/FBt3/detection_loss‚2.6044426461060843 epoch‚5 _runtime‚5425.314398305²ÕHu’à‚(:‡ ¥©ãÔ—±ž® gpu.0.gpu‚24.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚137.263' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.4526 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0WæpÌvá‚(:ë ¥©ãÔÆà‚± disk.rbd1.out‚ 14.6845703125' disk.nvme0c0n1.out‚28254.7998046875 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.0546875# memory_percent‚94.5255994796753 disk.rbd0.in‚0 disk.dm-0.in‚ 378.94140625 disk.sda2.out‚0 disk.drbd2265.in‚0% disk.nvme0n1.out‚28254.7998046875 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 7636.4375( proc.memory.availableMB‚ 896.92578125 disk.drbd2080.out‚0" disk.dm-0.out‚28254.7998046875 disk.sda3.in‚ 0.0546875 disk.sda.in‚ 0.0546875 disk.sda2.in‚0# proc.memory.rssMB‚ 2693.87109375* proc.memory.percent‚16.442084312438965 network.recv‚ 1.949088e+06% disk./.usageGB‚229.49884796142578 disk.drbd2265.out‚0 disk.rbd1.in‚0* disk./.usagePercent‚15.401860249050683 disk.drbd2266.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 378.94140625 disk.sda.out‚ 7636.4375 disk.dm-1.out‚ 7636.4375 proc.cpu.threads‚17 network.sent‚ 1.6465101e+07 disk.drbd2266.out‚0 disk.nvme0n1.in‚ 378.94140625 disk.drbd1716.out‚0 cpu‚25.346698039099294 disk.drbd2080.in‚04]Û’â‚(:‡ ´©ãÔò­‰® gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚133.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.6972 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0p°ÇÞ€ã‚(:õ ´©ãԋô° network.recv‚ 1.955078e+06$ disk.nvme0n1.out‚28256.060546875 disk.dm-1.in‚ 0.0546875 disk.dm-1.out‚ 7637.14453125 disk.drbd2266.out‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 378.94140625 disk.drbd1716.in‚0( proc.memory.availableMB‚ 897.41015625& disk.nvme0c0n1.out‚28256.060546875 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.sda3.in‚ 0.0546875 disk.rbd1.out‚ 14.6845703125 disk.sda.out‚ 7637.14453125 disk.drbd2265.in‚0 disk.rbd0.out‚0$ memory_percent‚94.52264308929443 network.sent‚ 1.6573713e+07% disk./.usageGB‚229.49882888793945 disk.dm-0.in‚ 378.94140625! disk.dm-0.out‚28256.060546875 disk.rbd1.in‚0 disk.drbd2080.in‚0* disk./.usagePercent‚15.401858969013013 disk.sda3.out‚ 7637.14453125 disk.nvme0n1.in‚ 378.94140625 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2693.87109375* proc.memory.percent‚16.442084312438965 disk.sda.in‚ 0.0546875 disk.drbd2266.in‚0 disk.drbd2265.out‚0 cpu‚25.345890298138393¦×ªÙä‚(:’ éãÔÚͳ® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚119.444' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.888800000000003 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°žÔ[Xå‚(:Í éãÔúü¡² disk.sda1.out‚0 network.sent‚ 1.6579943e+07 disk.nvme0n1.out‚28256.5 disk.drbd2265.in‚0 disk.dm-0.out‚28256.5 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 378.94921875) disk./.usagePercent‚15.40187023334452 disk.nvme0n1.in‚ 378.94921875 disk.sda3.in‚ 0.0546875 disk.sda3.out‚ 7637.6875 disk.rbd1.out‚ 14.6845703125& proc.memory.availableMB‚ 897.578125# proc.memory.rssMB‚ 2693.87109375* proc.memory.percent‚16.442084312438965 network.recv‚ 1.959624e+06 disk.drbd1716.in‚0 proc.cpu.threads‚17 disk.sda.in‚ 0.0546875 disk.dm-1.in‚ 0.0546875 disk.nvme0c0n1.out‚28256.5 cpu‚25.345108899088633 disk.drbd1716.out‚0 disk.dm-1.out‚ 7637.6875 disk.sda1.in‚0# memory_percent‚94.5216178894043 disk.sda.out‚ 7637.6875 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0% disk./.usageGB‚229.49899673461914 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 378.94921875 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0ŠxJÈ2æ- _wandbruntime‚5460  _runtime‚5460}·¢ÆçÀ _runtime‚ 5460.56948075 _step‚563 & lr-AdamW/pg1‚9.975657823627556e-05 & lr-AdamW/pg2‚9.975657823627556e-05 trainer/global_step‚6899 % _timestamp‚1.7884008359326303e+09)ãgËèÅ & lr-AdamW/pg1‚9.975657823627556e-05 & lr-AdamW/pg2‚9.975657823627556e-05 trainer/global_step‚6899 % _timestamp‚1.7884008359326303e+09 _runtime‚ 5460.56948075 _step‚563³oÔÇs2é- _wandbruntime‚5460  _runtime‚5460ÒjçÆ7êØO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 139345608 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚194113813409792 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚ 5460.57032297 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚92061479 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚184656807447552 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139325570 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90730147 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚194352947581526 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonito”Ž6§!r.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚203837765915136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90730147 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 139325570 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39105018 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚215578080828928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9892642705920 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 % _timestamp‚1.7884008359333172e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9695957973210 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚194113795406848 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205685654999040 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 139345608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9723970508288 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39104475 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205685678939136 trainer/global_step‚6899 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52957006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90730145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚194113795406848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 230055715 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9723970508288 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚184656807447552 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 139325570 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚90730147 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚194352947581526 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚203837765915136 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚90730147 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 139325570 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39105018 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚215578080828928 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9892642705920 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 % _timestamp‚1.7884008359333172e+09 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9695957973210 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚194113795406848 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚205685654999040 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 139345608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9723970508288 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39104475 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚205685678939136 trainer/global_step‚6899 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚52957006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚90730145 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚194113795406848 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 230055715 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9723970508288 J8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚194141064771584 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90742806 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚203866615434240 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚184682767669248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9725532659712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52964381 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚194141064771584 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9725532659712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39109706 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚215608406667264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244997120 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚203866615434240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 230088161 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230108227 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚194380279928614 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205714591887360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92074087 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9894059777024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚194141082774528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚194380097767642 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚184682750629888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199‚ùiR8îèM TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244997120 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚203866615434240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 230088161 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 230108227 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚194380279928614 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚205714591887360 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92074087 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9894059777024 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚194141082774528 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚194380097767642 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚184682750629888 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 230108227 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 230088161 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 139345357 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9697512259366 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚90742806 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 221057024 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚194141064771584 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚90742`<àÀ¶806 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚203866615434240 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚184682767669248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9725532659712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚52964381 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚194141064771584 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9725532659712 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39109706 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚215608406667264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992µW1•ô2ï- _wandbruntime‚5461  _runtime‚5461RX#¢WðÑ 1loss/FBt4/assignment_loss‚0.23515328782002187 .loss/FRt1/detection_loss‚2.167536108976398 % _timestamp‚1.7884008366059027e+09 /loss/FRt4/assignment_loss‚1.906352140155493 2loss/SRqqt1/assignment_loss‚1.4100341842008324 2loss/SRqqt2/assignment_loss‚1.7410105847079178 2loss/SRqqt3/assignment_loss‚1.3933146333797497 /loss/FRt4/detection_loss‚1.6670857146382332 .loss/FBt2/detection_loss‚2.787638150728666 /loss/FBt4/detection_loss‚0.6371688514341971 trainer/global_step‚6899 0loss/FBt2/assignment_loss‚0.3535502526806471 0loss/SRqqt2/detection_loss‚2.533880065227377 /loss/FBt1/detection_loss‚2.6773567335172133 _runtime‚5461.242627173 1loss/FBt1/assignment_loss‚0.29002108390943704 .loss/FRt2/detection_loss‚3.559785252525693 1loss/SRqqt1/detection_loss‚1.7268210855814128 /loss/FRt2/assignment_loss‚3.998744839713687 /loss/FRt3/assignment_loss‚2.251855414846669 0loss/FBt3/assignment_loss‚0.7223401591786219 1loss/SRqqt4/detection_loss‚1.2480919002777053 /loss/FRt1/assignment_loss‚3.074253817399343 1loss/SRqqt4/assignment_loss‚0.884544924906795 .loss/FBt3/detection_loss‚2.660930560185359 /loss/FRt3/detection_loss‚2.1487797764332397 1loss/SRqqt3/detection_loss‚1.8239566127353526 %loss/total_loss‚43.90020613516006 epoch‚5 _step‚566¬½D\ñÖ 2loss/SRqqt1/assignment_loss‚1.4100341842008324 2loss/SRqqt2/assignment_loss‚1.7410105847079178 2loss/SRqqt3/assignment_loss‚1.3933146333797497 /loss/FRt4/detection_loss‚1.6670857146382332 .loss/FBt2/detection_loss‚2.787638150728666 /loss/FBt4/detection_loss‚0.6371688514341971 trainer/global_step‚6899 0loss/FBt2/assignment_loss‚0.3535502526806471 0loss/SRqqt2/detection_loss‚2.533880065227377 /loss/FBt1/detection_loss‚2.6773567335172133 _runtime‚5461.242627173 1loss/FBt1/assignment_loss‚0.29002108390943704 .loss/FRt2/detection_loss‚3.559785252525693 1loss/SRqqt1/detection_loss‚1.7268210855814128 /loss/FRt2/assignment_loss‚3.998744839713687 /loss/FRt3/assignment_loss‚2.251855414846669 0loss/FBt3/assignment_loss‚0.7223401591786219 1loss/SRqqt4/detection_loss‚1.2480919002777053 /loss/FRt1/assignment_loss‚3.074253817399343 1loss/SRqqt4/assignment_loss‚0.884544924906795 .loss/FBt3/detection_loss‚2.660930560185359 /loss/FRt3/detection_loss‚2.1487797764332397 1loss/SRqqt3/detection_loss‚1.8239566127353526 %loss/total_loss‚43.90020613516006 epoch‚5 _step‚566 1loss/FBt4/assignment_loss‚0.23515328782002187 .loss/FRt1/detection_loss‚2.167536108976398 % _timestamp‚1.7884008366059027e+09 /loss/FRt4/assignment_loss‚1.906352140155493¶"“}ÿ›ò‚(: Ò©ãÔ°’»® gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚144.12' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.823999999999998 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ș…¶vó‚(:ë Ò©ãÔ¹ƒ•± disk.sda1.out‚0 disk.sda3.in‚ 0.0546875 disk.rbd1.in‚0 disk.sda.out‚ 7638.4453125% disk./.usageGB‚229.49894332885742 disk.drbd2265.out‚0 disk.dm-1.out‚ 7638.4453125 disk.dm-0.out‚28257.20703125& proc.memory.availableMB‚ 897.765625 proc.cpu.threads‚17* disk./.usagePercent‚15.401866649239041 disk.drbd2266.out‚0 disk.drbd2080.in‚0* proc.memory.percent‚16.442084312438965 network.sent‚ 1.6585337e+07 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.0546875 disk.rbd0.in‚0 disk.dm-0.in‚ 378.94921875" disk.nvme0c0n1.in‚ 378.94921875$ memory_percent‚94.52047348022461 disk.drbd2265.in‚0 disk.rbd0.out‚0 network.recv‚ 1.963355e+06 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0% disk.nvme0c0n1.out‚28257.20703125 disk.nvme0n1.in‚ 378.94921875# proc.memory.rssMB‚ 2693.87109375 disk.sda1.in‚0 disk.sda3.out‚ 7638.4453125 disk.rbd1.out‚ 14.6845703125# disk.nvme0n1.out‚28257.20703125 disk.sda.in‚ 0.0546875 cpu‚25.3442690745887 disk.sda2.in‚0 disk.sda2.out‚02Œ,"’ô‚(:‡ á©ãÔ’䉮 gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚134.022' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.8044 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OYÍ?rõ‚(:ç á©ãÔÞõ§²$ disk./.usageGB‚229.4989471435547 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.dm-0.in‚ 378.94921875 disk.rbd1.out‚ 14.6845703125 disk.sda.out‚ 7639.296875 cpu‚25.34345780210154 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.sda3.in‚ 0.0546875 disk.sda3.out‚ 7639.296875 disk.nvme0n1.in‚ 378.94921875 disk.sda.in‚ 0.0546875( proc.memory.availableMB‚ 897.27734375* proc.memory.percent‚16.442084312438965 network.sent‚ 1.6694009e+07 disk.rbd1.in‚0$ disk.nvme0c0n1.out‚ 28257.4453125* disk./.usagePercent‚15.401866905246573 disk.sda2.out‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.dm-0.out‚ 28257.4453125 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 378.94921875 proc.cpu.threads‚17 disk.dm-1.out‚ 7639.296875# proc.memory.rssMB‚ 2693.87109375 disk.sda2.in‚0 disk.dm-1.in‚ 0.0546875 disk.drbd2266.in‚0 network.recv‚ 1.969623e+06 disk.rbd0.out‚0" disk.nvme0n1.out‚ 28257.4453125$ memory_percent‚94.52345371246338ö8ö2ö- _wandbruntime‚5496  _runtime‚5496X¢Å÷¿ trainer/global_step‚6949 % _timestamp‚1.7884008714372673e+09 _runtime‚ 5496.0741226 _step‚567 & lr-AdamW/pg1‚9.820967435142665e-05 & lr-AdamW/pg2‚9.820967435142665e-05yw ÊøÄ _runtime‚ 5496.0741226 _step‚567 & lr-AdamW/pg1‚9.820967435142665e-05 & lr-AdamW/pg2‚9.820967435142665e-05 trainer/global_step‚6949 % _timestamp‚1.7884008714372673e+09·£µM2ù- _wandbruntime‚5496  _runtime‚5496ÌX¶‹Þ'úØO @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9793810830848 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9793810830848 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9793641830912 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚6949 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚205271092477440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚207131302410240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140338310 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39377570 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9963530278400 _step‚568 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140318272 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9765594913022 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚195477263643648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚5496.075011637 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚185953983603712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53325854 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚195477281646592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚205270905474560 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9793641830912 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚205271092477440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚195477281646592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚195477263643648 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39378114 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231701407 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚217094617909760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91363095 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53325856 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9765429791618 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚546 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 $ _timestamp‚1.788400871437992e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231681367 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91363097 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚205270905474560 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91363095 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140318272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9963315499520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚195719578516734 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚217094856628736 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91363097 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚185953966564352¯P›¬'ûÝO UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 140338310 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚205270905474560 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚91363095 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 140318272 _MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚ 9963315499520 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚195719578516734 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚217094856628736 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91363097 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚185953966564352 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9793810830848 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9793810830848 ZHDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚ 9793641830912 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 trainer/global_step‚6949 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚205271092477440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚207131302410240 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 140338310 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39377570 cQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚ 9963530278400 _step‚568 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 140318272 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9765594913022 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚195477263643648 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚5496.075011637 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚185953983603712 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53325854 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚195477281646592 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚205270905474560 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9793641830912 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚205271092477440 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚195477281646592 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚195477263643648 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚39378114 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 231701407 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚217094617909760 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91363095 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚53325856 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚544 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9765429791618 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚546 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 $ _timestamp‚1.788400871437992e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 231681367 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚91363097 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91375756 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚217124942051328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91375754 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185979909746688 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚195504533008384 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9795203982336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140338059 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884008721126168e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚205299754993664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚6949 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9795203982336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚207160215358464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9964726692864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231733879 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140358123 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚217125184951296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚195746910863822 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92716088 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9964945652736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚195504533008384 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91375754 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚205299567977472 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53333231 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚185979926786048 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚195504551011328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9766818956162 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231733879 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚195504551011328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91375756 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140358123 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53333233 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231713813 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚569 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚195746728702850 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5496.750001592 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408Ãk|Hî&þèM J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53333231 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚185979926786048 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚195504551011328 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9766818956162 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 231733879 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚195504551011328 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚91375756 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 140358123 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53333233 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 231713813 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 _step‚569 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚195746728702850 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚5496.750001592 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚39383380 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39382857 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚207160239298560 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9795034969088 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚92716613 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 140338059 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚205299754993664 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 231713813 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚91375756 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚217124942051328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚91375754 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚185979909746688 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚195504533008384 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9795203982336 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 140338059 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884008721126168e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚205299754993664 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚6949 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9795203982336 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚207160215358464 ]KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚ 9964726692864 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 231733879 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 140358123 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚217125184951296 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚195746910863822 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚92716088 aODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚ 9964945652736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚195504533008384 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚91375754 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚205299567977472 EDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚196840731880448 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚196840749883392 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9863482153472 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91996047 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚218611394250240 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93344171 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚6999 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93344711 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚208576973761536 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141310974 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 _runtime‚5532.023877318 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10034209904128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚196840731880448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚197086209451942 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚197086027291178 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141331012 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91996045 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚196840749883392 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9834901610026 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚218611159725568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚206704045033984 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9863313153536 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10034420488704 % _timestamp‚1.7884009073868885e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚206704045033984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9863482153472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53694704|/¢oæ'‹àO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 233327059 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39649467 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚187251125681152 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9835066731430 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10034420488704 % _timestamp‚1.7884009073868885e+09 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚206704045033984 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9863482153472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚53694704 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚196840731880448 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚196840749883392 [IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚ 9863482153472 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚91996047 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚218611394250240 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93344171 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 trainer/global_step‚6999 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚93344711 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚208576973761536 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 141310974 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 _runtime‚5532.023877318 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10034209904128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚196840731880448 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚197086209451942 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚197086027291178 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 141331012 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚91996045 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚196840749883392 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9834901610026 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚218611159725568 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚206704045033984 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 WEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚ 9863313153536 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚187277085902848 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9864875304960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92008706 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9864706291712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10035627524096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141330761 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233339465 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92008704 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚206732707536896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚196868019248128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93357513 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚218641722707968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141350825 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚196868001245184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚187277068863488 % _timestamp‚1.7884009080663602e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233359531 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚196868019248128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚206732894553088 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10035835998208 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 trainer/global_step‚6999 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚206732894553088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9836455896182 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚208605886709760 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9836290774570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233359531 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141350825 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _runtime‚5532.703561455 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 233339465 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93356979 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53702083 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0b—tª ŽêM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _runtime‚5532.703561455 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 233339465 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93356979 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚53702083 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚197113359638058 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚197113541799030 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92008706 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9864706291712 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 141330761 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚53702081 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39654898 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚208605862769664 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9864875304960 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚218641490293760 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚187277085902848 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9864875304960 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚92008706 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9864706291712 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10035627524096 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 141330761 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 233339465 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92008704 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚206732707536896 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚196868019248128 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93357513 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚532 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚218641722707968 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 141350825 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚196868001245184 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚187277068863488 % _timestamp‚1.7884009080663602e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 233359531 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚196868019248128 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚206732894553088 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 MCDevi›©Ú±FceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10035835998208 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 trainer/global_step‚6999 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚206732894553088 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9836455896182 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚208605886709760 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9836290774570 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 233359531 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 141350825 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408½àyXü2- _wandbruntime‚5532  _runtime‚5532ðAZÔ 2loss/SRqqt4/assignment_loss‚1.2170981448708158 0loss/FBt2/assignment_loss‚0.8068956086256662 %loss/total_loss‚46.72715372383632 epoch‚5 _runtime‚5532.704960996 /loss/SRqqt2/assignment_loss‚2.0884203838891 .loss/FRt4/detection_loss‚2.011839834973216 0loss/FRt2/assignment_loss‚4.2083517499268055 .loss/FBt3/detection_loss‚3.986123820145925 0loss/FRt1/assignment_loss‚3.2593118079165193 0loss/FRt4/assignment_loss‚1.8197478353977203 1loss/FBt3/assignment_loss‚0.11265527456998825 1loss/FBt4/assignment_loss‚0.19984912108551783 2loss/SRqqt1/assignment_loss‚1.9332825821289357 /loss/FRt1/detection_loss‚2.2530912997874806 0loss/SRqqt1/detection_loss‚2.014025978268461 /loss/FBt1/detection_loss‚2.7102341294288634 trainer/global_step‚6999 2loss/SRqqt3/assignment_loss‚1.2682008728125067 1loss/FBt1/assignment_loss‚0.27849680408789934 /loss/FRt2/detection_loss‚2.8148134667426348 1loss/SRqqt4/detection_loss‚1.2893304083157668 _step‚574 1loss/SRqqt3/detection_loss‚2.1298955342797345 % _timestamp‚1.7884009080682132e+09 /loss/FRt3/assignment_loss‚2.355760129722389 .loss/FRt3/detection_loss‚2.353318805630143 1loss/SRqqt2/detection_loss‚2.7303331946504525 /loss/FBt2/detection_loss‚2.2700629383325577 /loss/FBt4/detection_loss‚0.6160139982472174l^:é_‘Ù 0loss/FRt2/assignment_loss‚4.2083517499268055 .loss/FBt3/detection_loss‚3.986123820145925 0loss/FRt1/assignment_loss‚3.2593118079165193 0loss/FRt4/assignment_loss‚1.8197478353977203 1loss/FBt3/assignment_loss‚0.11265527456998825 1loss/FBt4/assignment_loss‚0.19984912108551783 2loss/SRqqt1/assignment_loss‚1.9332825821289357 /loss/FRt1/detection_loss‚2.2530912997874806 0loss/SRqqt1/detection_loss‚2.014025978268461 /loss/FBt1/detection_loss‚2.7102341294288634 trainer/global_step‚6999 2loss/SRqqt3/assignment_loss‚1.2682008728125067 1loss/FBt1/assignment_loss‚0.27849680408789934 /loss/FRt2/detection_loss‚2.8148134667426348 1loss/SRqqt4/detection_loss‚1.2893304083157668 _step‚574 1loss/SRqqt3/detection_loss‚2.1298955342797345 % _timestamp‚1.7884009080682132e+09 /loss/FRt3/assignment_loss‚2.355760129722389 .loss/FRt3/detection_loss‚2.353318805630143 1loss/SRqqt2/detection_loss‚2.7303331946504525 /loss/FBt2/detection_loss‚2.2700629383325577 /loss/FBt4/detection_loss‚0.6160139982472174 2loss/SRqqt4/assignment_loss‚1.2170981448708158 0loss/FBt2/assignment_loss‚0.8068956086256662 %loss/total_loss‚46.72715372383632 epoch‚5 _runtime‚5532.704960996 /loss/SRqqt2/assignment_loss‚2.0884203838891 .loss/FRt4/detection_loss‚2.011839834973216¾Ñ"yš’‚(:’ ŽªãÔû¶® gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚129.949' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.989800000000002 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßY›Ü“‚(:ô ŽªãÔ”óܱ disk.sda1.out‚0 disk.rbd1.in‚0' disk.nvme0c0n1.out‚28259.2998046875 disk.nvme0n1.in‚ 378.94921875 disk.sda.out‚ 7640.8671875# proc.memory.rssMB‚ 2693.87109375* proc.memory.percent‚16.442084312438965* disk./.usagePercent‚15.401880985660958 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.0546875( proc.memory.availableMB‚ 897.04296875 cpu‚25.341043315994945% disk.nvme0n1.out‚28259.2998046875 disk.sda2.in‚0 disk.dm-1.out‚ 7640.8671875 disk.drbd2265.out‚0 network.sent‚ 1.6814353e+07 network.recv‚ 1.983408e+06 disk.sda3.in‚ 0.0546875 disk.sda3.out‚ 7640.8671875" disk.nvme0c0n1.in‚ 378.94921875 disk.drbd2266.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 14.6845703125 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 378.94921875" disk.dm-0.out‚28259.2998046875 disk.sda.in‚ 0.0546875 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0$ memory_percent‚94.52488422393799$ disk./.usageGB‚229.4991569519043 disk.sda1.in‚0Ì~’”‚(:‡ ªãÔѳò­ gpu.0.gpu‚35.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚120.262' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0524 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø]%¼‚•‚(:÷ ªãÔæôݱ disk.sda1.in‚0" disk.nvme0c0n1.in‚ 378.94921875( proc.memory.availableMB‚ 897.05078125* proc.memory.percent‚16.442084312438965 disk.drbd1716.in‚0& disk.nvme0n1.out‚28261.39404296875 disk.sda.out‚ 7641.4765625 disk.sda2.out‚0 disk.dm-1.in‚ 0.0546875 disk.sda1.out‚0 disk.nvme0n1.in‚ 378.94921875$ memory_percent‚94.52483654022217 disk.drbd1716.out‚0 disk.drbd2266.out‚0 cpu‚25.34026018587127 disk.sda3.out‚ 7641.4765625 disk.sda2.in‚0 disk.dm-1.out‚ 7641.4765625 proc.cpu.threads‚17 network.sent‚ 1.6923137e+07 network.recv‚ 1.989849e+06 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 378.94921875 disk.sda3.in‚ 0.0546875 disk.drbd2266.in‚0* disk./.usagePercent‚15.401991324908215 disk.drbd2265.in‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚28261.39404296875# proc.memory.rssMB‚ 2693.87109375% disk./.usageGB‚229.50080108642578 disk.drbd2265.out‚0 disk.rbd0.in‚0# disk.dm-0.out‚28261.39404296875 disk.rbd1.out‚ 14.6845703125 disk.sda.in‚ 0.0546875Nt>é’–‚(:‡ ¬ªãÔóÌî gpu.0.gpu‚33.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚131.841' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.3682 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ýŽÌH„—‚(:ù ¬ªãÔ«ÊÙ°( disk.nvme0c0n1.out‚28263.14599609375& proc.memory.availableMB‚ 897.046875 network.recv‚ 1.994073e+06 disk.dm-0.in‚ 378.94921875 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0# disk.dm-0.out‚28263.14599609375 disk.sda3.in‚ 0.0546875* proc.memory.percent‚16.442084312438965 cpu‚25.339490935098638% disk./.usageGB‚229.50080108642578 disk.nvme0n1.in‚ 378.94921875$ memory_percent‚94.52486038208008 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 14.6845703125 disk.sda.out‚ 7641.94140625 disk.sda2.in‚0 network.sent‚ 1.6929023e+07 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 378.94921875& disk.nvme0n1.out‚28263.14599609375 disk.dm-1.in‚ 0.0546875 proc.cpu.threads‚17 disk.sda2.out‚0 disk.drbd1716.out‚0* disk./.usagePercent‚15.401991324908215 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda3.out‚ 7641.94140625# proc.memory.rssMB‚ 2693.87109375 disk.drbd2265.out‚0 disk.sda.in‚ 0.0546875 disk.dm-1.out‚ 7641.94140625 disk.drbd2266.in‚0–39Õ2˜- _wandbruntime‚5567  _runtime‚5567Áä#çÆ™À $ _timestamp‚1.788400943155559e+09 _runtime‚5567.792490668 _step‚575 & lr-AdamW/pg1‚9.512199263626776e-05 & lr-AdamW/pg2‚9.512199263626776e-05 trainer/global_step‚70499G6˚Š$ _timestamp‚1.788400943155559e+09 _runtime‚5567.792490668 _step‚575 & lr-AdamW/pg1‚9.512199263626776e-05 & lr-AdamW/pg2‚9.512199263626776e-05 trainer/global_step‚7049¿âD Ÿ2›- _wandbruntime‚5567  _runtime‚5567µ]Jà'œÚO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚208137371596288 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 142303676 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚198204218120192 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 142303676 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54063554 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 _runtime‚ 5567.79333947 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚220127756401664 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚188548301837312 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54063556 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚208137371596288 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 _step‚576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 234932671 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚548 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142323714 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9904373428434 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39921104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚208137184593408 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93984658 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10105369753600 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚198452840387150 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚92628997 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚208137184593408 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚546 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚210022597232640 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ^LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚ 9933153476096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54063556 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚208137371596288 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 _step‚576 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 234932671 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚548 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 142323714 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9904373428434 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚39921104 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚208137184593408 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚93984658 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10105369753600 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚198452840387150 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0Àΰ+¬2ž- _wandbruntime‚5568  _runtime‚5568uµ ŸåM O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚7049 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 234985183 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9934377614336 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142343527 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92641656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚210051510180864 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 234985183 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92641654 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚208166034112512 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92641654 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚220158086698496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10106576517632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚198231469481984 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 % _timestamp‚1.7884009438335078e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142343527 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142323463 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 142323463 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39926512 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9905927714590 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9934546627584 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9934377614336 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚208165847096320 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 234965117 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚208166034112512 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93997443 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9905762592978/KzKð& êM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚198231487484928 \JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚ 9934546627584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚536 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54070933 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚93997979 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54070931 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚198231469481984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 234985183 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚92641654 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚208166034112512 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚92641654 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚220158086698496 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10106576517632 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚198231469481984 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 % _timestamp‚1.7884009438335078e+09 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 142343527 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 142323463 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 142323463 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚39926512 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9905927714590 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 YGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚ 9934546627584 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 UCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚ 9934377614336 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚208165847096320 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 234965117 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚208166034112512 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚93997443 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9905762592978 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚7049 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 234985183 XFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚ 9934377614336 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 142343527 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚92641656 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚210051510180864 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143316416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40191478 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143316416 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 % _timestamp‚1.7884009788768308e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54432404 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93261947 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 143296378 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143296378 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚199567686356992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 236578363 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚209570324152832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54432406 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚221644583640064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚580 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚209570511155712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚211468268583936 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚189845443914752 _runtime‚5603.513842946 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93261945 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94623884 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚199567686356992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9973845246842 trainer/global_step‚7099 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 236558323 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94623339 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211468244643840 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10176104471552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚199567668354048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚221644349115392 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9974010368246 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚189845460954112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 L?DeviceStatsMonitßO]¨#or.on_train_batch_start/active.large_pool.freed‚93261945 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚209570324152832 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93261947 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚199819289161594 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0îìê'­äO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚189845460954112 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93261945 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚209570324152832 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93261947 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚199819289161594 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10002655798784 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚199819471322358 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚209570511155712 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10002655798784 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 143316416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40191478 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 143316416 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 % _timestamp‚1.7884009788768308e+09 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54432404 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93261947 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 143296378 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 143296378 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚199567686356992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 236578363 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚209570324152832 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚54432406 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚221644583640064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _step‚580 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚209570511155712 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚211468268583936 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚189845443914752 _runtime‚5603.513842946 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93261945 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚94623884 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚199567686356992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ZHDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚ 9973845246842 trainer/global_step‚7099 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 236558323 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚94623339 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚211468244643840 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10176104471552 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚199567668354048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚221644349115392 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ^LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚ 9974010368246 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040Ä‘CŸ2®- _wandbruntime‚5604  _runtime‚5604¸ ê ï&¯éM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚209599173671936 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10177521419264 _runtime‚5604.182995063 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 236610835 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211497157592064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9975399532998 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚199594955721728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚221674679011328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 143336229 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚209599173671936 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚211497181532160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚529 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚189871404136448 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94636574 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚581 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143336229 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10177734087680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9975234411386 % _timestamp‚1.7884009795455203e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93274606 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54439783 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚199846803669446 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10004217950208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 236590769 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40196791 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40196264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94636045 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54439781 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143316165 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚199594937718784 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 236590769 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚529 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚189871404136448 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚94636574 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚581 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 143336229 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10177734087680 XFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚ 9975234411386 % _timestamp‚1.7884009795455203e+09 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚93274606 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚54439783 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚199846803669446 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10004217950208 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 236590769 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40196791 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40196264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚94636045 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚54439781 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 143316165 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚199594937718784 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 236590769 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10177521419264 _runtime‚5604.182995063 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 236610835 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚211497157592064 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚ 9975399532998 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚199594955721728 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚221674679011328 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 143336229 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚209599173671936 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚211497181532160 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40462695 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54801254 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93894897 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144289080 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204293120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144289080 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95263396 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚212913892055040 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _runtime‚5639.216018007 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚211003650715136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚200931154593792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144309118 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚211003463712256 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144309118 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚211003463712256 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚201185920096802 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚191142603031552 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚200931136590848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40462142 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚555 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10043317065250 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚223161127906816t ;ë'½åO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚555 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10043317065250 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚223161127906816 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 $ _timestamp‚1.788401014579024e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10072327121408 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10043482186654 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 trainer/global_step‚7149 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚93894897 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚553 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚200931154593792 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚200931136590848 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚212913915995136 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚93894895 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚93894895 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚223160899673600 _step‚584 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 238183975 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10072327121408 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚191142620070912 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 238204015 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40462695 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚54801254 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚93894897 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 144289080 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204293120 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 144289080 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95263396 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚212913892055040 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _runtime‚5639.216018007 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚211003650715136 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚200931154593792 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 144309118 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚211003463712256 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 144309118 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚211003463712256 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚201185920096802 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚191142603031552 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚200931136590848 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40462142 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040È•èœ&¾‚(:’ ÷ªãÔé˜ý­ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚121.088' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.217599999999997 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\Ôá—‚¿‚(:÷ ÷ªãÔü¯Ü° disk.sda2.in‚0* proc.memory.percent‚16.442084312438965 network.recv‚ 2.019038e+06* disk./.usagePercent‚15.401952155755474# disk.dm-0.out‚28265.16943359375 disk.sda1.out‚0 disk.sda.in‚ 0.0546875 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 378.94921875 disk.sda2.out‚0 disk.dm-1.in‚ 0.0546875 disk.drbd2265.out‚0$ proc.memory.availableMB‚896.6875 disk.rbd1.in‚0# proc.memory.rssMB‚ 2693.87109375 cpu‚25.335613947540697 disk.dm-0.in‚ 378.94921875 disk.sda3.in‚ 0.0546875" disk.nvme0c0n1.in‚ 378.94921875 disk.drbd1716.in‚0$ memory_percent‚94.52705383300781% disk./.usageGB‚229.50021743774414 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.rbd1.out‚ 14.6845703125& disk.nvme0n1.out‚28265.16943359375 disk.drbd1716.out‚0 proc.cpu.threads‚17 network.sent‚ 1.7164083e+07 disk.sda1.in‚0 disk.sda3.out‚ 7644.49609375( disk.nvme0c0n1.out‚28265.16943359375 disk.sda.out‚ 7644.49609375 disk.dm-1.out‚ 7644.49609375 disk.drbd2266.out‚0 disk.drbd2265.in‚0ºáq2À- _wandbruntime‚5640  _runtime‚5640=’òÁêM SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40467452 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚211032126215168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238236487 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 trainer/global_step‚7149 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚507 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93907554 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚200958423958528 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95276083 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144328931 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10248633315328 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10044706229794 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _runtime‚ 5640.11727962 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10073889272832 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚93907556 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚223191462258688 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238216421 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚212942828943360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93907554 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚585 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚191168546213888 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10073720259584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228219904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10044871351406 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 238216421 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚212942828943360 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚93907554 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 _step‚585 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚191168546213888 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10073720259584 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228219904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10044871351406 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40467452 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚211032126215168 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 238236487 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 trainer/global_step‚7149 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚507 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚93907554 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚200958423958528 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95276083 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 144328931 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10248633315328 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10044706229794 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _runtime‚ 5640.11727962 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10073889272832 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199É`Íx;2Ã- _wandbruntime‚5640  _runtime‚5640*6ç•YÄÓ .loss/FBt3/detection_loss‚2.958208841936929 trainer/global_step‚7149 % _timestamp‚1.7884010154815998e+09 .loss/FRt1/detection_loss‚2.089876521870777 0loss/FRt1/assignment_loss‚3.6949035379121886 /loss/FBt3/assignment_loss‚0.415203014238963 1loss/SRqqt3/detection_loss‚2.0361385114338932 1loss/SRqqt4/detection_loss‚1.0356851136180716 epoch‚5 1loss/FBt2/assignment_loss‚0.23469268282254063 /loss/FRt4/detection_loss‚1.7610328899962562 2loss/SRqqt2/assignment_loss‚1.6706539231442636 2loss/SRqqt4/assignment_loss‚1.1422103928782465 0loss/FRt3/assignment_loss‚2.7105900627725266 2loss/SRqqt3/assignment_loss‚1.6814161673264236 .loss/FRt2/detection_loss‚2.935657534748316 /loss/FRt3/detection_loss‚2.6565389817251877 0loss/SRqqt2/detection_loss‚2.590318674041379 /loss/FRt2/assignment_loss‚3.570689793676138 0loss/FRt4/assignment_loss‚1.4920384596501077 1loss/FBt1/assignment_loss‚0.35066239091042917 .loss/FBt2/detection_loss‚2.445446267724037 %loss/total_loss‚44.73028468465524 _runtime‚5640.118327417 0loss/FBt4/assignment_loss‚0.1949772603388745 1loss/SRqqt1/assignment_loss‚2.102382374237533 /loss/FBt4/detection_loss‚0.6957434549830951 _step‚586 1loss/SRqqt1/detection_loss‚1.6281894663417782 /loss/FBt1/detection_loss‚2.6370283663272858°ßú^ÅØ 2loss/SRqqt2/assignment_loss‚1.6706539231442636 2loss/SRqqt4/assignment_loss‚1.1422103928782465 0loss/FRt3/assignment_loss‚2.7105900627725266 2loss/SRqqt3/assignment_loss‚1.6814161673264236 .loss/FRt2/detection_loss‚2.935657534748316 /loss/FRt3/detection_loss‚2.6565389817251877 0loss/SRqqt2/detection_loss‚2.590318674041379 /loss/FRt2/assignment_loss‚3.570689793676138 0loss/FRt4/assignment_loss‚1.4920384596501077 1loss/FBt1/assignment_loss‚0.35066239091042917 .loss/FBt2/detection_loss‚2.445446267724037 %loss/total_loss‚44.73028468465524 _runtime‚5640.118327417 0loss/FBt4/assignment_loss‚0.1949772603388745 1loss/SRqqt1/assignment_loss‚2.102382374237533 /loss/FBt4/detection_loss‚0.6957434549830951 _step‚586 1loss/SRqqt1/detection_loss‚1.6281894663417782 /loss/FBt1/detection_loss‚2.6370283663272858 .loss/FBt3/detection_loss‚2.958208841936929 trainer/global_step‚7149 % _timestamp‚1.7884010154815998e+09 .loss/FRt1/detection_loss‚2.089876521870777 0loss/FRt1/assignment_loss‚3.6949035379121886 /loss/FBt3/assignment_loss‚0.415203014238963 1loss/SRqqt3/detection_loss‚2.0361385114338932 1loss/SRqqt4/detection_loss‚1.0356851136180716 epoch‚5 1loss/FBt2/assignment_loss‚0.23469268282254063 /loss/FRt4/detection_loss‚1.7610328899962562ÊŒ|»‘Æ‚(:’ †«ãÔ¼£½® gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚143.469' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.693799999999996 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0›]¨rÇ‚(:ç †«ãÔˆ‚ü± disk.rbd0.out‚0* disk./.usagePercent‚15.402077599447253* proc.memory.percent‚16.442084312438965 disk.dm-1.out‚ 7644.9375 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 7644.9375 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 378.94921875$ disk.nvme0n1.out‚28265.689453125 network.recv‚ 2.023812e+06 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.sda1.in‚0$ memory_percent‚94.52569484710693( proc.memory.availableMB‚ 896.91015625 network.sent‚ 1.7259976e+07$ disk./.usageGB‚229.5020866394043 disk.drbd2080.out‚0 disk.rbd1.out‚ 14.6845703125 proc.cpu.threads‚17 disk.sda.out‚ 7644.9375 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.dm-0.in‚ 378.94921875 disk.drbd2266.out‚0& disk.nvme0c0n1.out‚28265.689453125 disk.nvme0n1.in‚ 378.94921875 cpu‚25.33484817789761 disk.sda.in‚ 0.0546875 disk.drbd1716.in‚0! disk.dm-0.out‚28265.689453125 disk.sda3.in‚ 0.0546875# proc.memory.rssMB‚ 2693.87109375 disk.dm-1.in‚ 0.0546875 disk.drbd2080.in‚0SÀ¾È‚(:’ •«ãÔàò‡® gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚142.643' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.528599999999997 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œqnF~É‚(:ó •«ãÔùíÁ± disk.drbd2265.in‚0 disk.drbd2080.in‚0! disk.dm-0.out‚28265.919921875 disk.sda3.in‚ 0.0546875" disk.nvme0c0n1.in‚ 378.94921875 disk.sda.in‚ 0.0546875 disk.dm-1.out‚ 7645.42578125 disk.rbd0.in‚0 disk.dm-0.in‚ 378.94921875 disk.rbd1.out‚ 14.6845703125 disk.sda2.out‚0$ memory_percent‚94.52569484710693( proc.memory.availableMB‚ 896.91015625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 378.94921875 disk.sda2.in‚0 network.recv‚ 2.030055e+06 disk.sda.out‚ 7645.42578125* disk./.usagePercent‚15.402077599447253$ disk./.usageGB‚229.5020866394043 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 7645.42578125# proc.memory.rssMB‚ 2693.87109375* proc.memory.percent‚16.442084312438965 network.sent‚ 1.7358068e+07 disk.dm-1.in‚ 0.0546875 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 cpu‚25.33405820153024 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚28265.919921875& disk.nvme0c0n1.out‚28265.919921875 disk.drbd1716.in‚0Ýz2Ê- _wandbruntime‚5675  _runtime‚5675I!/XÅË¿ _step‚587 % lr-AdamW/pg1‚9.05106072826597e-05 % lr-AdamW/pg2‚9.05106072826597e-05 trainer/global_step‚7199 % _timestamp‚1.7884010505469596e+09 _runtime‚5675.183789626¿§ ÊÌÄ _runtime‚5675.183789626 _step‚587 % lr-AdamW/pg1‚9.05106072826597e-05 % lr-AdamW/pg2‚9.05106072826597e-05 trainer/global_step‚7199 % _timestamp‚1.7884010505469596e+09Ë&»~2Í- _wandbruntime‚5675  _runtime‚5675âÃç'ÎáO L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94527845 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10141998444032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10142167443968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 trainer/global_step‚7199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚214359539466240 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55170104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94527847 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚212436603271680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚202294622830592 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10317937891840 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚212436790274560 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚212436790274560 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚192439779187712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚192439762148352 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94527847 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚224677711882752 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 _runtime‚5675.184750689 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55170106 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10112788883658 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚202294604827648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7884010505476444e+09 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 239809627 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚202294604827648 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40732955 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145301820 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 145281782 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚202552733192774 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚202552551032010 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚202294622830592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10141998444032 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚212436603271680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40733503 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10318148476416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95903059 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145281782 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _step‚588 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 #óÕÏæO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10318148476416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚95903059 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 145281782 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _step‚588 N@DeviceStatsMonitor.bö´à"on_train_batch_start/active_bytes.all.current‚ 187002880 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚94527845 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10141998444032 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10142167443968 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 trainer/global_step‚7199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚214359539466240 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55170104 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚94527847 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚212436603271680 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚202294622830592 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10317937891840 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚212436790274560 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚212436790274560 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚192439779187712 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚192439762148352 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚94527847 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚224677711882752 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 _runtime‚5675.184750689 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚55170106 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10112788883658 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚202294604827648 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7884010505476444e+09 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 239809627 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚202294604827648 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚40732955 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 145301820 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 145281782 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚202552733192774 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚202552551032010 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚202294622830592 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10141998444032 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚212436603271680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚40733503 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95916059 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40739083 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚507 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145321633 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145301569 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚224707806522368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10114343169814 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10114178048202 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 239842073 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95916566 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 239842073 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚212465265774592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94540504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚214388476354560 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10319354107904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚505 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚202321874192384 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚202321874192384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10319568873472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚7199 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚212465452790784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10143560595456 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55177483 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚224708045228032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 239862139 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚202321892195328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10143391582208 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 239862139 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94540506 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55177481 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7884010512228146e+09 _step‚589 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40738578 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚214388452414464 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚202580065539862 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94540506 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚192465705330688 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10143391582208]QÉ@ö&ÒðM C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚94540504 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚214388476354560 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10319354107904 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚505 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚202321874192384 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚202321874192384 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10319568873472 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚7199 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚212465452790784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10143560595456 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55177483 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚224708045228032 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 239862139 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚202321892195328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10143391582208 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 239862139 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚94540506 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55177481 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7884010512228146e+09 _step‚589 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚40738578 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚214388452414464 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚202580065539862 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚94540506 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚192465705330688 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10143391582208 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚212465452790784 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚94540504 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚202579883378890 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚192465722370048 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚95916059 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚40739083 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚507 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 145321633 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 145301569 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚224707806522368 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10114343169814 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10114178048202 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 239842073 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚95916566 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 239842073 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚212465265774592 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1Í­ä¨2Ó- _wandbruntime‚5675  _runtime‚5675’¶7ÿ\ÔÖ 2loss/SRqqt4/assignment_loss‚1.0690443342661156 0loss/FBt1/assignment_loss‚0.3763210073941284 1loss/SRqqt2/detection_loss‚2.7900154228721346 trainer/global_step‚7199 0loss/FRt4/assignment_loss‚1.6957990733476787 .loss/FBt3/detection_loss‚2.920357210295541 _runtime‚5675.861062493 0loss/FRt2/assignment_loss‚3.9167605748302057 2loss/SRqqt2/assignment_loss‚2.3261766991178905 .loss/FRt2/detection_loss‚2.891739357458918 /loss/FRt3/detection_loss‚2.2340843313270144 1loss/FBt4/assignment_loss‚0.10196854422870397 /loss/FBt2/detection_loss‚2.9001703688076566 _step‚590 1loss/SRqqt1/assignment_loss‚1.630068740658462 /loss/FRt4/detection_loss‚1.8343033796677797 0loss/FRt3/assignment_loss‚2.5793926570150587 /loss/FRt1/detection_loss‚2.1698987531661995 1loss/SRqqt1/detection_loss‚1.8287516650557518 1loss/SRqqt3/detection_loss‚1.8216907172178736 /loss/FBt4/detection_loss‚0.5206800342562634 %loss/total_loss‚44.74209459939112 0loss/FRt1/assignment_loss‚3.5217472195625303 1loss/SRqqt4/detection_loss‚1.0216286575750393 epoch‚5 % _timestamp‚1.7884010512243257e+09 2loss/SRqqt3/assignment_loss‚1.7842407655351016 0loss/FBt2/assignment_loss‚0.3442033380270004 0loss/FBt3/assignment_loss‚0.1105915520872145 .loss/FBt1/detection_loss‚2.352460195620855xá?NaÕÛ 2loss/SRqqt3/assignment_loss‚1.7842407655351016 0loss/FBt2/assignment_loss‚0.3442033380270004 0loss/FBt3/assignment_loss‚0.1105915520872145 .loss/FBt1/detection_loss‚2.352460195620855 2loss/SRqqt4/assignment_loss‚1.0690443342661156 0loss/FBt1/assignment_loss‚0.3763210073941284 1loss/SRqqt2/detection_loss‚2.7900154228721346 trainer/global_step‚7199 0loss/FRt4/assignment_loss‚1.6957990733476787 .loss/FBt3/detection_loss‚2.920357210295541 _runtime‚5675.861062493 0loss/FRt2/assignment_loss‚3.9167605748302057 2loss/SRqqt2/assignment_loss‚2.3261766991178905 .loss/FRt2/detection_loss‚2.891739357458918 /loss/FRt3/detection_loss‚2.2340843313270144 1loss/FBt4/assignment_loss‚0.10196854422870397 /loss/FBt2/detection_loss‚2.9001703688076566 _step‚590 1loss/SRqqt1/assignment_loss‚1.630068740658462 /loss/FRt4/detection_loss‚1.8343033796677797 0loss/FRt3/assignment_loss‚2.5793926570150587 /loss/FRt1/detection_loss‚2.1698987531661995 1loss/SRqqt1/detection_loss‚1.8287516650557518 1loss/SRqqt3/detection_loss‚1.8216907172178736 /loss/FBt4/detection_loss‚0.5206800342562634 %loss/total_loss‚44.74209459939112 0loss/FRt1/assignment_loss‚3.5217472195625303 1loss/SRqqt4/detection_loss‚1.0216286575750393 epoch‚5 % _timestamp‚1.7884010512243257e+09ÎuEºýÖ‚(:’ ¤«ãÔ‹Ä® gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚135.089' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.017799999999998 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þG´Þׂ(:ó ¤«ãÔ™åÝ° disk.drbd1716.out‚0 disk.dm-1.in‚ 0.0546875 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 378.94921875& disk.nvme0n1.out‚28266.44677734375* proc.memory.percent‚16.442084312438965# proc.memory.availableMB‚896.375 network.recv‚ 2.034241e+06 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚28266.44677734375 disk.nvme0n1.in‚ 378.94921875* disk./.usagePercent‚15.402077599447253 disk.dm-1.out‚ 7646.31640625 disk.drbd2080.in‚0# proc.memory.rssMB‚ 2693.87109375 disk.sda.in‚ 0.0546875 disk.drbd2080.out‚0 disk.dm-0.in‚ 378.94921875# disk.dm-0.out‚28266.44677734375 disk.sda1.out‚0ö~«y disk.rbd1.out‚ 14.6845703125 cpu‚25.33336291786989 proc.cpu.threads‚17 network.sent‚ 1.736401e+07 disk.sda2.in‚0 disk.sda3.out‚ 7646.31640625$ memory_percent‚94.52896118164062$ disk./.usageGB‚229.5020866394043 disk.sda.out‚ 7646.31640625 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.sda3.in‚ 0.0546875<ôhØ‚(:’ ³«ãÔϘƒ® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚149.908' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.981599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0JPZÙ‚(:ô ³«ãÔúÀ‚² disk.drbd1716.out‚0 disk.drbd2080.in‚0" disk.nvme0c0n1.in‚ 378.94921875 disk.sda2.in‚0 disk.dm-1.in‚ 0.0546875 disk.dm-1.out‚ 7646.671875 disk.dm-0.in‚ 378.94921875' proc.memory.availableMB‚ 896.0546875 disk.sda.out‚ 7646.671875 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 7646.671875 disk.rbd1.in‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 14.6845703125( disk.nvme0c0n1.out‚28266.71240234375 disk.nvme0n1.in‚ 378.94921875$ memory_percent‚94.53091621398926 cpu‚25.332564976209635 network.recv‚ 2.040846e+06* disk./.usagePercent‚15.402078879484923 disk.sda.in‚ 0.0546875 disk.drbd2265.in‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17 network.sent‚ 1.7472937e+07# disk.dm-0.out‚28266.71240234375 disk.sda1.out‚0& disk.nvme0n1.out‚28266.71240234375# proc.memory.rssMB‚ 2693.87109375 disk.sda3.in‚ 0.0546875* proc.memory.percent‚16.442084312438965 disk.sda2.out‚0 disk.drbd2266.in‚0% disk./.usageGB‚229.50210571289062‡.Æk2Ú- _wandbruntime‚5710  _runtime‚5710Ù+¶ÇÛÁ % _timestamp‚1.7884010861588223e+09 _runtime‚5710.795655759 _step‚591 & lr-AdamW/pg1‚8.898013069729189e-05 & lr-AdamW/pg2‚8.898013069729189e-05 trainer/global_step‚7249ù5eÌÜÆ % _timestamp‚1.7884010861588223e+09 _runtime‚5710.795655759 _step‚591 & lr-AdamW/pg1‚8.898013069729189e-05 & lr-AdamW/pg2‚8.898013069729189e-05 trainer/global_step‚7249Ï|ï¬q2Ý- _wandbruntime‚5710  _runtime‚5710“=Ìç'ÞáO E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚203658073064448 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 241455319 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213869742831104 _step‚592 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95160795 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146274484 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 241435279 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚213869929833984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55538954 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10389045184512 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213869929833984 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95160797 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚203658091067392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10211838766592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41005767 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230330368 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206390272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚193736921265152 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 trainer/global_step‚7249 _runtime‚5710.796517565 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10182260702066 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213869742831104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚203658091067392 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚203919181967218 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146274484 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10211669766656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884010861594977e+09 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10388838794240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚203658073064448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95160797 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146294522 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚215805186877440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10388838794240 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚203658073064448 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚95160797 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 146294522 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚215805186877440 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚213869742831104 _step‚592 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚95160795 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 146274484 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 241435279 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚213869929833984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚55538954 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10389045184512 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚213869929833984 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚95160797 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚203658091067392 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10211838766592 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41005767 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230330368 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206390272 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚193736921265152 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 trainer/global_step‚7249 _runtime‚5710.796517565 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10182260702066 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚213869742831104 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚203658091067392 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚203919181967218 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 146274484 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10211669766656 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 % _timestamp‚1.7884010861594977e+09Ðèr1o2à- _wandbruntime‚5711  _runtime‚5711ìz.ê"&áêM ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10390253261312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 241487791 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚226224589695488 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241487791 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚203685342429184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚203946514314098 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146314335 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 241467725 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41010639 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10183814988222 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95173454 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10213062904832 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96556970 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41011179 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146314335 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚193762881486848 _runtime‚5711.478207594 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95173454 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55546333 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚215834123765760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚215834099825664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55546331 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 trainer/global_step‚7249 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213898592350208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95173456 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚203946696475070 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146294271 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10213231918080 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _step‚593 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚193762864447488 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 241467725 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 $ _timestamp‚1.788401086841034e+09 T@DeviceStatsMonitor.on_train_baÞ&3Îtch_end/active_bytes.all.allocated‚213898592350208 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985=´©õ&âïM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _step‚593 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚193762864447488 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 241467725 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 $ _timestamp‚1.788401086841034e+09 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚213898592350208 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10390253261312 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 241487791 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚226224589695488 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 241487791 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚203685342429184 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚203946514314098 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 146314335 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 241467725 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41010639 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10183814988222 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚95173454 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10213062904832 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚96556970 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚540 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚41011179 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 146314335 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚193762881486848 _runtime‚5711.478207594 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚95173454 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚55546333 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚215834123765760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚215834099825664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚55546331 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 trainer/global_step‚7249 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚213898592350208 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚95173456 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚542 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚203946696475070 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 146294271 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10213231918080 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0Ñœ¨%2ã- _wandbruntime‚5711  _runtime‚5711šûEZäÔ 0loss/FBt2/assignment_loss‚0.4569409921764411 0loss/FBt3/assignment_loss‚0.7758359061554074 _runtime‚5711.479262023 1loss/FBt4/assignment_loss‚0.14823184241795406 /loss/FRt1/detection_loss‚2.2683270268662032 /loss/FRt4/detection_loss‚1.8272801358252764 .loss/FBt2/detection_loss‚2.999090778827667 2loss/SRqqt1/assignment_loss‚1.8841967675834894 2loss/SRqqt3/assignment_loss‚1.4813582959865768 0loss/FRt1/assignment_loss‚2.9567022129546765 /loss/FRt3/detection_loss‚2.1439666506648063 epoch‚5 trainer/global_step‚7249 1loss/SRqqt2/assignment_loss‚2.185449844492333 2loss/SRqqt4/assignment_loss‚0.9623619174285803 1loss/SRqqt1/detection_loss‚2.1858823779556484 1loss/SRqqt2/detection_loss‚2.4774492255278995 1loss/SRqqt3/detection_loss‚1.5712042458000637 %loss/total_loss‚45.33707234992499 /loss/FBt4/detection_loss‚0.8547825759438716 /loss/FRt2/assignment_loss‚3.886080503463745 0loss/FBt1/assignment_loss‚0.4138198451438673 1loss/SRqqt4/detection_loss‚0.8761906525419082 $ _timestamp‚1.788401086842503e+09 /loss/FRt4/assignment_loss‚2.041763131817182 /loss/FRt2/detection_loss‚2.8769666807992116 /loss/FBt1/detection_loss‚2.2486281792322793 .loss/FBt3/detection_loss‚3.025968164205551 _step‚594 0loss/FRt3/assignment_loss‚2.7885943961143496‡÷P_åÙ /loss/FRt2/assignment_loss‚3.886080503463745 0loss/FBt1/assignment_loss‚0.4138198451438673 1loss/SRqqt4/detection_loss‚0.8761906525419082 $ _timestamp‚1.788401086842503e+09 /loss/FRt4/assignment_loss‚2.041763131817182 /loss/FRt2/detection_loss‚2.8769666807992116 /loss/FBt1/detection_loss‚2.2486281792322793 .loss/FBt3/detection_loss‚3.025968164205551 _step‚594 0loss/FRt3/assignment_loss‚2.7885943961143496 0loss/FBt2/assignment_loss‚0.4569409921764411 0loss/FBt3/assignment_loss‚0.7758359061554074 _runtime‚5711.479262023 1loss/FBt4/assignment_loss‚0.14823184241795406 /loss/FRt1/detection_loss‚2.2683270268662032 /loss/FRt4/detection_loss‚1.8272801358252764 .loss/FBt2/detection_loss‚2.999090778827667 2loss/SRqqt1/assignment_loss‚1.8841967675834894 2loss/SRqqt3/assignment_loss‚1.4813582959865768 0loss/FRt1/assignment_loss‚2.9567022129546765 /loss/FRt3/detection_loss‚2.1439666506648063 epoch‚5 trainer/global_step‚7249 1loss/SRqqt2/assignment_loss‚2.185449844492333 2loss/SRqqt4/assignment_loss‚0.9623619174285803 1loss/SRqqt1/detection_loss‚2.1858823779556484 1loss/SRqqt2/detection_loss‚2.4774492255278995 1loss/SRqqt3/detection_loss‚1.5712042458000637 %loss/total_loss‚45.33707234992499 /loss/FBt4/detection_loss‚0.8547825759438716Òtj[‘æ‚(:† «ãÔÐò®® gpu.0.gpu‚26.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚122.395' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.479 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÍLø€ç‚(:õ «ãÔ¾ý…² disk.sda1.in‚0 disk.sda.in‚ 0.0546875 disk.dm-0.in‚ 378.94921875 disk.drbd2080.in‚0 network.recv‚ 2.044314e+06 disk.sda1.out‚0 disk.sda.out‚ 7647.1640625 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.rbd1.out‚ 14.6845703125 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.0546875# proc.memory.rssMB‚ 2693.87109375 network.sent‚ 1.7478277e+07 disk.nvme0n1.in‚ 378.94921875* proc.memory.percent‚16.442084312438965 cpu‚25.332697430713928" disk.nvme0c0n1.in‚ 378.94921875' disk.nvme0c0n1.out‚28268.4697265625 disk.rbd0.out‚0( proc.memory.availableMB‚ 896.27734375 disk.drbd2080.out‚0 disk.sda3.in‚ 0.0546875 disk.rbd1.in‚0 disk.dm-1.out‚ 7647.1640625 disk.drbd2265.in‚0 disk.drbd2266.out‚0 disk.sda3.out‚ 7647.1640625% disk.nvme0n1.out‚28268.4697265625 disk.sda2.in‚0" disk.dm-0.out‚28268.4697265625$ memory_percent‚94.52955722808838 disk.drbd2265.out‚0 proc.cpu.threads‚17* disk./.usagePercent‚15.402080159522596% disk./.usageGB‚229.502124786376952!Éb‘è‚(:† Ñ«ãÔÑš‰® gpu.0.gpu‚8.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚151.023' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.2046 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Äôé‚(:ô Ñ«ãÔää± disk.sda3.out‚ 7647.63671875 disk.sda.in‚ 0.0546875$ memory_percent‚94.53105926513672 cpu‚25.344387136870083 network.recv‚ 2.050406e+06 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2693.87109375 network.sent‚ 1.758695e+07 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0% proc.memory.availableMB‚ 896.03125 proc.cpu.threads‚17' disk.nvme0c0n1.out‚28268.7314453125 disk.sda2.out‚0 disk.dm-1.in‚ 0.0546875 disk.dm-1.out‚ 7647.63671875 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 378.94921875 disk.sda3.in‚ 0.0546875 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 14.7705078125" disk.nvme0c0n1.in‚ 378.94921875 disk.nvme0n1.in‚ 378.94921875* disk./.usagePercent‚15.402081695567801% disk./.usageGB‚229.50214767456055 disk.sda.out‚ 7647.63671875* proc.memory.percent‚16.442084312438965" disk.dm-0.out‚28268.7314453125 disk.sda1.out‚0 disk.rbd1.in‚0% disk.nvme0n1.out‚28268.7314453125¡¬×òê‚(:… à«ãÔ檊® gpu.0.gpu‚42.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚123.51' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.702 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚09Þ+ë‚(:ö à«ãÔþ–Ú± disk.nvme0n1.in‚ 378.94921875& disk.nvme0n1.out‚28269.49755859375 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0* disk./.usagePercent‚15.402080415530131% disk./.usageGB‚229.50212860107422 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0% proc.memory.availableMB‚ 896.03125 proc.cpu.threads‚17 disk.sda.in‚ 0.0546875 disk.rbd0.out‚0 disk.rbd1.out‚ 14.8330078125 disk.dm-0.in‚ 378.94921875 disk.sda1.in‚0 disk.rbd1.in‚0# proc.memory.rssMB‚ 2693.87109375* proc.memory.percent‚16.442084312438965# disk.dm-0.out‚28269.49755859375 disk.sda3.out‚ 7648.48046875" disk.nvme0c0n1.in‚ 378.94921875 disk.dm-1.out‚ 7648.48046875 disk.rbd0.in‚0 network.sent‚ 1.7593207e+07 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 network.recv‚ 2.054744e+06( disk.nvme0c0n1.out‚28269.49755859375 disk.sda2.out‚0 disk.drbd2265.out‚0$ memory_percent‚94.53105926513672 cpu‚25.3573141497062 disk.sda3.in‚ 0.0546875 disk.sda.out‚ 7648.48046875 disk.dm-1.in‚ 0.0546875W3½’ì‚(:‡ ï«ãÔ¦’°¯ gpu.0.gpu‚21.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚122.154' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚08ãÆ„í‚(:ù ï«ãÔÄ—± disk.rbd1.out‚ 14.8330078125" disk.nvme0c0n1.in‚ 378.94921875 disk.sda.out‚ 7648.94140625* proc.memory.percent‚16.442084312438965 disk.dm-1.out‚ 7648.94140625 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda3.in‚ 0.0546875& disk.nvme0n1.out‚28269.72802734375 disk.sda2.out‚0 disk.drbd1716.out‚0 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.0546875 disk.drbd1716.in‚0 cpu‚25.369226903711613 network.recv‚ 2.059239e+06* disk./.usagePercent‚15.402084255643143 disk.drbd2265.out‚0 disk.sda3.out‚ 7648.94140625 network.sent‚ 1.759934e+07$ disk./.usageGB‚229.5021858215332 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 378.94921875$ memory_percent‚94.53403949737549 disk.sda2.in‚0( proc.memory.availableMB‚ 895.54296875 disk.rbd0.in‚0# disk.dm-0.out‚28269.72802734375 disk.drbd2266.in‚0 disk.dm-0.in‚ 378.94921875( disk.nvme0c0n1.out‚28269.72802734375# proc.memory.rssMB‚ 2693.87109375 disk.dm-1.in‚ 0.0546875 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.sda1.in‚0ñýâK2î- _wandbruntime‚5762  _runtime‚5762<]*>M*ïÇT ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚26 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2460 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 146738816 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚10235925281792 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚217 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 LFDeviceStatsMonitor.on_validation_batch_start/active.large_pool.current‚2 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚199 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚416 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚204718831994880 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 $ _timestamp‚1.788401137540013e+09 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚10235797973504 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2460 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 5041553408 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 145311232 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚95748980 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚204718831994880 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 115961344 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 5041553408 OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 242503773 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚41263809 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 139901440 GADeviceStatsMonitor.on_validation_batch_start/num_sync_all_streams‚2 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚199 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚985 [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚10235925281792 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚10413299144704 _step‚595 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 242487794 TNDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 269877760 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 4624220160 QIDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚416 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚214954757276672 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚55894339 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚41263525 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚214954611965440 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 242503773 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 146738816 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚97157864 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 140964952 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 4757245257 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚95748978 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚204950108388674 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 123925592 _runtime‚5762.177262389 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 4912020992 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚95748980 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚95748978 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 410712064 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚284 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 146754793 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚286 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚416 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚214954611965440 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚227323509514752 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚204950249353626 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚55894341 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚216910326331392 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚10413183183360 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚217 trainer/global_step‚7253 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚214954757276672 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 410712064 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 145311232 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 4912020992 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 5041553408 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 242487794 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚10206295082306 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚204718813991936 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚97158150 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚416 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚214954757276672 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚55894339 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚41263525 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚214954611965440 SEDeviceStatsMonitor.on_validation_batch_start/allocation.all.allocated‚ 242503773 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 146738816 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚97157864 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 140964952 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 4757245257 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 QDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚95748978 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚204950108388674 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 123925592 _runtime‚5762.177262389 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 4912020992 YLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚95748980 UHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚95748978 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 ZLDeviceSÊ \q7 tatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 410712064 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚284 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.allocated‚ 146754793 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚286 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚416 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚214954611965440 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚227323509514752 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚204950249353626 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚55894341 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚216910326331392 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚10413183183360 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚217 trainer/global_step‚7253 [GDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.allocated‚214954757276672 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 410712064 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 145311232 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 4912020992 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 5041553408 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 242487794 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚10206295082306 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚204718813991936 VIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚97158150 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚214973971598336 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚204968357548674 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚217 _step‚596 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 115961344 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 242521881 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 4912020992 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 5041553408 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚10236274417664 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚214973971598336 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚10236147109376 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚214973826287104 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 % _timestamp‚1.7884011384984896e+09 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 146762189 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 410712064 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚204737679177728 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 242521881 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 146746212 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 406789969 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚97169378 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚416 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 242505902 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 5041553408 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚227343602117632 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚204737679177728 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 242505902 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚199 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚55900839 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 4624220160 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 146762189 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚227343462216192 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 410712064 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚55900841 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚217 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 140964952 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚10206766614234 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 145311232 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚284 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 127308288 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚10236147109376 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚217 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚95759692 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚199 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚95759690 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚985 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚214973826287104 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2460 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 139901440 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚216929953837056 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚10413648280576 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚10413532319232 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚10236274417664 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚194761714860032 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚10206642688642 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 417333248 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 5041553408 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚227343602117632 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚204737679177728 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 242505902 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚199 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚55900839 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 4624220160 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 146762189 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚227343462216192 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 410712064 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚55900841 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚217 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 140964952 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚10206766614234 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 145311232 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚284 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 127308288 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚10236147109376 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚217 SFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚95759692 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚199 SFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚95759690 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚985 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚214973826287104 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2460 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 139901440 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚216929953837056 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚10413648280576 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚10413532319232 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚10236274417664 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚194761714860032 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚10206642688642 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 417333248 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 5041553408 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚214973971598336 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚204968357548674 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚217 _step‚596 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 115961344 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 242521881 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 4912020992 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 5041553408 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚10236274417664 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚214973971598336 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚10236147109376 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚214973826287104 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 % _timestamp‚1.7884011384984896e+09 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 146762189 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 410712064 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚204737679177728 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 242521881 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 146746212 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 406789969 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚97169378 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚416 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 242505902 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 5041553408Ô¢8-3‘ô‚(:† þ«ãÔñâ³® gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚122.154' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0|2¨ƒõ‚(:ø þ«ãÔÈâʲ disk.drbd1716.out‚0 disk.dm-1.in‚ 0.0546875 cpu‚25.38114774323894 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.out‚ 7649.54296875 disk.rbd1.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 7649.54296875 disk.drbd2266.in‚0 network.sent‚ 1.7604733e+07 disk.rbd0.out‚0 disk.rbd1.out‚ 14.865234375( disk.nvme0c0n1.out‚28273.10498046875 disk.drbd2266.out‚0$ disk./.usageGB‚229.5049057006836* disk./.usagePercent‚15.402266789015057 disk.drbd2265.out‚0$ memory_percent‚94.53127384185791 disk.sda3.in‚ 0.0546875 disk.nvme0n1.in‚ 378.94921875 disk.sda.out‚ 7649.54296875 disk.drbd1716.in‚0# proc.memory.rssMB‚ 2693.87109375# disk.dm-0.out‚28273.10498046875 network.recv‚ 2.062583e+06 disk.drbd2265.in‚0 disk.rbd0.in‚0( proc.memory.availableMB‚ 895.99609375 disk.drbd2080.out‚0 disk.dm-0.in‚ 378.94921875 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 378.94921875& disk.nvme0n1.out‚28273.10498046875 disk.sda.in‚ 0.0546875 disk.sda2.in‚0* proc.memory.percent‚16.442084312438965 disk.drbd2080.in‚0}Í[‹2ö- _wandbruntime‚5775  _runtime‚5775œ£w3d÷µ§ =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,jet/accuracy_4_of_5‚0.008064515888690948 (particle/f_score‚0.34102165699005127 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2676631510257721 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12943081557750702 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02313566952943802 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.49366575479507446 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8181818127632141 particle/accuracy_6_of_7‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.026072483509778976 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11396761238574982 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5263158679008484 jet/accuracy_3_of_7‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.970622181892395 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.16349877417087555 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.37865501642227173 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_3_of_5‚0.0914810448884964 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05476190894842148 /"!W³ëùPurity/0FRt0SRqqt3FBt/event_purity‚0.140625 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3675924837589264 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15348035097122192 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970667362213135 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9759185314178467 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.032894738018512726 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9214926362037659 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9885268807411194 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4587755799293518 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13913708925247192 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.307407408952713 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12462969869375229 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5528929829597473 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.48720452189445496 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9895312786102295 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0851503238081932 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ."Purity/3FRt0SRqqt1FBt/event_purity‚0.09375 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.029012402519583702 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 +jet/accuracy_3_of_4‚0.07230715453624725 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 )loss/val_total_loss‚45.07688022091346 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9219858646392822 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.022431600838899612 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.19700752198696136 particle/accuracy_3_of_6‚1 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/3FRt0SRqqt*FBt/event_purity‚0.0277777798473835 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3792145848274231 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5277777910232544 jet/accuracy_6_of_6‚0 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4375 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.01245275977998972 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.23536035418510437 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5682691931724548 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025109948590397835 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.25 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.550622284412384 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.09555556625127792 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1968391090631485 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9864341616630554 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.022270116955041885 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 jet/accuracy_7_of_7‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1487690955400467 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13780827820301056 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1403227150440216 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 epoch‚5 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9773584604263306 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12437351793050766 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 + Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_7‚1 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02083333395421505 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5523122549057007 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20715835690498352 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1818181872367859 +jet/accuracy_3_of_6‚0.11538461595773697 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011890497989952564 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5039682984352112 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17604872584342957 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 (particle/accuracy‚0.8399575352668762 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10638298094272614 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.40630438923835754 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.18045534193515778 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11409053951501846 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.46875 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5654314160346985 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2482939213514328 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5112735629081726 particle/accuracy_4_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15715306997299194 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3020438849925995 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.25 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4066323935985565 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24281609058380127 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.365670382976532 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4221491515636444 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9973264932632446 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.47812992334365845 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5175066590309143 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14779545366764069 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 particle/accuracy_3_of_7‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17995963990688324 +jet/accuracy_2_of_2‚0.10499948263168335 +particle/specificity‚0.9462683796882629 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4495760202407837 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 7validation_average_jet_accuracy‚0.42980560660362244 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9878025650978088 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3912900388240814 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03874767944216728 _step‚597 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.248697891831398 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16137532889842987 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19718456268310547 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06722067296504974 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2807539403438568 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.27755919098854065 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17408521473407745 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.54774010181427 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.503451943397522 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 jet/accuracy_1_of_7‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.0722101479768753 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24538037180900574 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18111686408519745 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40376049280166626 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1894736886024475 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3623511791229248 <"Purity/0FRt3SRqqt0FBt/event_purity‚0.0049019609577953815 particle/accuracy_7_of_7‚0.5 particle/accuracy_1_of_4‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.17438271641731262 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31648850440979004 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010002944618463516 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 particle/accuracy_4_of_6‚1 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.003225806402042508 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.21085858345031738 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_3‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.961403489112854 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35324347019195557 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5301328897476196 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.43103450536727905 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 *jet/accuracy_1_of_2‚0.7408992648124695 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.40823638439178467 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9619252681732178 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5151345729827881 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5683630108833313 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13252472877502441 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4933915436267853 particle/accuracy_2_of_2‚1 * Purity/*FRt3SRqqt1FBt/FRt_purity‚0.375 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01899641565978527 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0038699398282915354 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5184492468833923 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 *jet/accuracy_1_of_4‚0.9116920232772827 jet/accuracy_4_of_6‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.48290228843688965 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5001165270805359 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.007036290131509304 jet/accuracy_5_of_6‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9951314330101013 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17309609055519104 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9899340271949768 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.8181818127632141 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5211309790611267 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.971230149269104 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9907218813896179 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6394803524017334 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 particle/accuracy_2_of_4‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_7‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2791843116283417 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5167227387428284 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5178796052932739 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.13218390941619873 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37511396408081055 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 particle/accuracy_3_of_5‚1 9"Purity/3FRt*SRqqt1FBt/event_purity‚0.0416666679084301 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2507246434688568 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9923167824745178 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4224867820739746 particle/accuracy_5_of_5‚1 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4496716260910034 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.48295077681541443 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4129636287689209 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.537151038646698 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.449160635471344 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16395017504692078 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9828237891197205 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13865827023983002 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9877693057060242 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13895706832408905 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6830701231956482 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 trainer/global_step‚7253 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9908296465873718 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08098159730434418 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_3‚0.2362530380487442 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5226442813873291 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.42283952236175537 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4560185670852661 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12545619904994965 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7"Purity/0FRt2SRqqt*FBt/event_purity‚0.06352549046278 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 particle/accuracy_2_of_7‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1487690955400467 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 particle/accuracy_2_of_6‚1 particle/accuracy_5_of_6‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4481475055217743 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5132526755332947 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9869836568832397 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9791561365127563 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8258887529373169 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9846290946006775 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3586561381816864 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1428571492433548 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735631346702576 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4095969796180725 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4395851194858551 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5882513523101807 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.375 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.36778274178504944 +jet/accuracy_2_of_4‚0.34932881593704224 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.47922590374946594 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5955762267112732 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1111111119389534 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.372688889503479 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.35798609256744385 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5112735629081726 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15981613099575043 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.38723722100257874 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20441654324531555 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4642857015132904 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.38619527220726013 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_3‚0.8486443758010864 particle/accuracy_1_of_3‚1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.49235719442367554 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 +particle/sensitivity‚0.2654821276664734 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.003225806402042508 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5243840217590332 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5388888716697693 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3194444477558136 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9969386458396912 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.017231008037924767 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5290805101394653 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9813601970672607 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973471164703369 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.36764705181121826 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5310074090957642 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 jet/accuracy_5_of_7‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 _runtime‚5775.996091869 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 5 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.50670325756073 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.025743260979652405 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.37249746918678284 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.45094063878059387 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3458113372325897 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5066666603088379 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.096385657787323 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportüüœb_ ion‚0.0016121031949296594 *jet/accuracy_2_of_6‚0.5064102411270142 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20400063693523407 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 <"Purity/0FRt3SRqqt*FBt/event_purity‚0.0049019609577953815 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4375 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4783288836479187 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2194419801235199 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3030303120613098 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.36187365651130676 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5424780249595642 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9710884690284729 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4377002716064453 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.29549792408943176 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_3_of_4‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.47642597556114197 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3750000298023224 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 jet/accuracy_1_of_6‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5434232950210571 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23499156534671783 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.007806401699781418 particle/accuracy_3_of_3‚1 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.47333332896232605 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3194444477558136 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5412188768386841 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.008944318629801273 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12194720655679703 particle/accuracy_1_of_1‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 particle/accuracy_5_of_7‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9910286664962769 jet/accuracy_4_of_7‚0 % _timestamp‚1.7884011513566911e+09XVÐu“røº§ & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.40823638439178467 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9619252681732178 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5151345729827881 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5683630108833313 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13252472877502441 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.4933915436267853 particle/accuracy_2_of_2‚1 * Purity/*FRt3SRqqt1FBt/FRt_purity‚0.375 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 :"Purity/*FRt1SRqqt2FBt/event_purity‚0.01899641565978527 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0038699398282915354 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5184492468833923 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 *jet/accuracy_1_of_4‚0.9116920232772827 jet/accuracy_4_of_6‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 :"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.48290228843688965 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5001165270805359 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,jet/accuracy_4_of_4‚0.007036290131509304 jet/accuracy_5_of_6‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9951314330101013 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.17309609055519104 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9899340271949768 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.8181818127632141 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5211309790611267 8 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.20000000298023224 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.971230149269104 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9907218813896179 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6394803524017334 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 particle/accuracy_2_of_4‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 jet/accuracy_2_of_7‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2791843116283417 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5167227387428284 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5178796052932739 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.13218390941619873 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 jet/accuracy_6_of_7‚0 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.37511396408081055 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 particle/accuracy_3_of_5‚1 9"Purity/3FRt*SRqqt1FBt/event_purity‚0.0416666679084301 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.2507246434688568 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.25 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9923167824745178 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4224867820739746 particle/accuracy_5_of_5‚1 7 Purity/0FRt3SRqqt1FBt/FBt_purity‚0.8888888955116272 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4496716260910034 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.48295077681541443 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4129636287689209 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.537151038646698 8"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.449160635471344 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16395017504692078 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/*FRt3SRqqt*FBt/FBt_purity‚0.9642857313156128 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9828237891197205 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.13865827023983002 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.4821428656578064 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9877693057060242 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.13895706832408905 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6830701231956482 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 trainer/global_step‚7253 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9908296465873718 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08098159730434418 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_2_of_3‚0.2362530380487442 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5226442813873291 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 :"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.42283952236175537 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.4560185670852661 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.12545619904994965 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 7"Purity/0FRt2SRqqt*FBt/event_purity‚0.06352549046278 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 particle/accuracy_2_of_7‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1487690955400467 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 particle/accuracy_2_of_6‚1 particle/accuracy_5_of_6‚1 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4481475055217743 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5132526755332947 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9869836568832397 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9791561365127563 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8258887529373169 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9846290946006775 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.3586561381816864 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 particle/accuracy_1_of_2‚1 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.1666666716337204 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1428571492433548 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735631346702576 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4095969796180725 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4395851194858551 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5882513523101807 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.46875 * Purity/2FRt*SRqqt2FBt/FRt_purity‚0.375 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.36778274178504944 +jet/accuracy_2_of_4‚0.34932881593704224 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.47922590374946594 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5955762267112732 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.1111111119389534 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.372688889503479 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.35798609256744385 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5112735629081726 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.15981613099575043 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.38723722100257874 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20441654324531555 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4642857015132904 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.38619527220726013 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_3‚0.8486443758010864 particle/accuracy_1_of_3‚1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 :"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.49235719442367554 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 +particle/sensitivity‚0.2654821276664734 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ;"Purity/*FRt3SRqqt*FBt/event_purity‚0.003225806402042508 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5243840217590332 :"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5388888716697693 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_4_of_4‚1 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3194444477558136 *"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9969386458396912 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.017231008037924767 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 *"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.5 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5290805101394653 8 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.47058823704719543 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9813601970672607 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973471164703369 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.36764705181121826 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5310074090957642 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 jet/accuracy_5_of_7‚0 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 _runtime‚5775.996091869 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 5 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.50670325756073 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.4444444477558136 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.025743260979652405 7 Purity/0FRt3SRqqt*FBt/FBt_purity‚0.8888888955116272 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.37249746918678284 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 :"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.45094063878059387 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3458113372325897 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/*FRt3SRqqt1FBt/FBt_purity‚0.9629629850387573 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.5066666603088379 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 8"Purity/*FRt1SRqqt1FBt/event_purity‚0.096385657787323 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 *jet/accuracy_2_of_6‚0.5064102411270142 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.20400063693523407 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 <"Purity/0FRt3SRqqt*FBt/event_purity‚0.0049019609577953815 -"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.4375 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4783288836479187 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2194419801235199 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3030303120613098 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.36187365651130676 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5424780249595642 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9710884690284729 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4377002716064453 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.29549792408943176 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 particle/accuracy_3_of_4‚1 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 :"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.47642597556114197 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3750000298023224 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 jet/accuracy_1_of_6‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5434232950210571 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23499156534671783 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ;"Purity/3FRt*SRqqt*FBt/event_purity‚0.007806401699781418 particle/accuracy_3_of_3‚1 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.47333332896232605 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3194444477558136 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5412188768386841 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.008944318629801273 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.12194720655679703 particle/accuracy_1_of_1‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.40625 particle/accuracy_5_of_7‚1 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9910286664962769 jet/accuracy_4_of_7‚0 % _timestamp‚1.7884011513566911e+09 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,jet/accuracy_4_of_5‚0.008064515888690948 (particle/f_score‚0.34102165699005127 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2676631510257721 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.12943081557750702 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.02313566952943802 :"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.49366575479507446 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8181818127632141 particle/accuracy_6_of_7‚1 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.026072483509778976 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.11396761238574982 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.5263158679008484 jet/accuracy_3_of_7‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 6 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.970622181892395 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.16349877417087555 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.37865501642227173 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 *jet/accuracy_3_of_5‚0.0914810448884964 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.05476190894842148 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.140625 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3675924837589264 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.15348035097122192 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9970667362213135 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9759185314178467 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.011627906933426857 ;"Purity/2FRt0SRqqt2FBt/event_purity‚0.032894738018512726 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9214926362037659 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9885268807411194 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4587755799293518 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.13913708925247192 6 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.307407408952713 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.12462969869375229 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5528929829597473 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 :"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.48720452189445496 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9895312786102295 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt*SRqqt*FBt/event_purity‚0.0851503238081932 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ."Purity/3FRt0SRqqt1FBt/event_purity‚0.09375 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1944444626569748 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 ;"Purity/*FRt2SRqqt0FBt/event_purity‚0.029012402519583702 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 +jet/accuracy_3_of_4‚0.07230715453624725 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 )loss/val_total_loss‚45.07688022091346 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9219858646392822 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.022431600838899612 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.19700752198696136 particle/accuracy_3_of_6‚1 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚1 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 9"Purity/3FRt0SRqqt*FBt/event_purity‚0.0277777798473835 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.3792145848274231 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 particle/accuracy_4_of_5‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.5277777910232544 jet/accuracy_6_of_6‚0 -"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.4375 :"Purity/*FRt2SRqqt1FBt/event_purity‚0.01245275977998972 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.23536035418510437 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_5‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5682691931724548 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.025109948590397835 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.25 8"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.550622284412384 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.09555556625127792 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1968391090631485 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9864341616630554 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.022270116955041885 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 jet/accuracy_7_of_7‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1487690955400467 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.13780827820301056 7 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.1403227150440216 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 +"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.25 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 epoch‚5 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9773584604263306 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.12437351793050766 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 + Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 particle/accuracy_1_of_7‚1 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02083333395421505 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05±jˆÈ.! ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5523122549057007 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.20715835690498352 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1818181872367859 +jet/accuracy_3_of_6‚0.11538461595773697 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.011890497989952564 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5039682984352112 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1944444626569748 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.17604872584342957 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 (particle/accuracy‚0.8399575352668762 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10638298094272614 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.40630438923835754 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.18045534193515778 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11409053951501846 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.46875 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5654314160346985 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2482939213514328 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5112735629081726 particle/accuracy_4_of_7‚1 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15715306997299194 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3020438849925995 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.25 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4066323935985565 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.24281609058380127 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.365670382976532 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4221491515636444 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9973264932632446 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.47812992334365845 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5175066590309143 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14779545366764069 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 particle/accuracy_3_of_7‚1 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.17995963990688324 +jet/accuracy_2_of_2‚0.10499948263168335 +particle/specificity‚0.9462683796882629 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4495760202407837 8 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.44736841320991516 7validation_average_jet_accuracy‚0.42980560660362244 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9878025650978088 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3912900388240814 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.03874767944216728 _step‚597 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.248697891831398 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16137532889842987 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.19718456268310547 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.06722067296504974 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2807539403438568 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.27755919098854065 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.17408521473407745 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.54774010181427 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.503451943397522 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 6 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.903508722782135 jet/accuracy_1_of_7‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 9"Purity/0FRt2SRqqt0FBt/event_purity‚0.0722101479768753 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24538037180900574 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18111686408519745 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.40376049280166626 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1894736886024475 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3623511791229248 <"Purity/0FRt3SRqqt0FBt/event_purity‚0.0049019609577953815 particle/accuracy_7_of_7‚0.5 particle/accuracy_1_of_4‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.17438271641731262 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 8 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.31648850440979004 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010002944618463516 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 particle/accuracy_4_of_6‚1 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ;"Purity/*FRt3SRqqt0FBt/event_purity‚0.003225806402042508 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.21085858345031738 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 particle/accuracy_2_of_3‚1 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.961403489112854 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35324347019195557 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5301328897476196 :"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.43103450536727905 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 *jet/accuracy_1_of_2‚0.7408992648124695 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25ÕnŽG"’ù‚(:‡ ¬ãÔ£³¢® gpu.0.gpu‚44.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚121.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2658 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—°Öxú‚(:í ¬ãÔœèƱ network.recv‚ 2.068838e+06 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 1.7713029e+07' disk.nvme0c0n1.out‚28396.4169921875 disk.dm-1.in‚ 0.0546875 disk.dm-1.out‚ 7649.98046875 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 378.94921875 disk.sda1.in‚0# memory_percent‚94.5270299911499* disk./.usagePercent‚15.402297509919165 disk.rbd0.in‚0" disk.dm-0.out‚28396.4169921875 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 378.94921875 disk.sda1.out‚0 disk.sda3.in‚ 0.0546875 disk.rbd1.out‚ 14.865234375 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2265.out‚0% disk./.usageGB‚229.50536346435547 disk.nvme0n1.in‚ 378.94921875% disk.nvme0n1.out‚28396.4169921875 disk.sda2.in‚0 disk.drbd1716.out‚0( proc.memory.availableMB‚ 896.69140625( proc.memory.percent‚16.4398193359375 disk.sda3.out‚ 7649.98046875 disk.drbd2080.out‚0 disk.sda.in‚ 0.0546875 disk.sda.out‚ 7649.98046875 disk.drbd2266.out‚0 disk.drbd2265.in‚0 proc.memory.rssMB‚2693.5 cpu‚25.380861813724607¶ k{’û‚(:‡ œ¬ãÔ”ôª® gpu.0.gpu‚34.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚121.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2658 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0®ˆÐwü‚(:ì œ¬ãÔÑõ° disk.rbd1.in‚0& disk.nvme0n1.out‚28520.55419921875# disk.dm-0.out‚28520.55419921875( proc.memory.percent‚16.4398193359375 disk.sda1.in‚0 disk.sda3.out‚ 7650.578125 disk.sda.out‚ 7650.578125 disk.dm-1.out‚ 7650.578125 disk.drbd2266.out‚0 disk.drbd2265.in‚0" disk.nvme0c0n1.in‚ 378.94921875 disk.nvme0n1.in‚ 378.94921875 disk.sda.in‚ 0.0546875 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.0546875 disk.rbd0.in‚0$ memory_percent‚94.52855587005615 cpu‚25.379973398325493$ disk./.usageGB‚229.5048942565918 disk.drbd2265.out‚0 disk.rbd0.out‚0* disk./.usagePercent‚15.402266020992453 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 378.94921875( proc.memory.availableMB‚ 896.44140625 proc.cpu.threads‚17 network.sent‚ 1.7837253e+07 network.recv‚ 2.074641e+06 disk.sda3.in‚ 0.0546875 disk.rbd1.out‚14.92822265625 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.sda2.in‚0 proc.memory.rssMB‚2693.5( disk.nvme0c0n1.out‚28520.55419921875 disk.drbd2266.in‚0ºdL2ý- _wandbruntime‚5810  _runtime‚5810eÊÅÇþÁ trainer/global_step‚7299 % _timestamp‚1.7884011859501052e+09 _runtime‚5810.586940055 _step‚598 & lr-AdamW/pg1‚8.745351223007982e-05 & lr-AdamW/pg2‚8.745351223007982e-05UÝG|ÌÿÆ & lr-AdamW/pg1‚8.745351223007982e-05 & lr-AdamW/pg2‚8.745351223007982e-05 trainer/global_step‚7299 % _timestamp‚1.7884011859501052e+09 _runtime‚5810.586940055 _step‚598Ö­Ö' 2€- _wandbruntime‚5810  _runtime‚5810 1êç'áO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚206210066009088 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚196161895302144 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10303522639872 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41570419 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚206210048006144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147735071 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚504 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228967714890752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚196161878262784 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96468601 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚206210048006144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147755109 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚218485858919424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚599 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56317304 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96468601 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97887221 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚206435709250568 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚216513588648960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10303353639936 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚206435527089804 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚216513401646080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚206210066009088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚216513401646080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚206210066009088 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚41570419 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚206210048006144 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 147735071 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚504 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚228967714890752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚196161878262784 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚96468601 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚206210048006144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 147755109 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚218485858919424 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚599 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚56317304 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚96468601 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚97887221 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚206435709250568× ªÞQ2ƒ- _wandbruntime‚5811  _runtime‚58110å¬h„ëM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDe—Ö-‰&viceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚228998041511424 trainer/global_step‚7299 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97899810 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56324681 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚216542251165184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10483269643776 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚196187821445120 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244236118 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147774922 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56324683 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10483480215040 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96481260 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96481260 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚218514795807744 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 % _timestamp‚1.7884011866187131e+09 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 _runtime‚5811.256045033 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚216542064148992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10275203113176 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228998276022784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚206463041597656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚196187838484480 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10304746778112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147774922 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚218514771867648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚206237335373824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96481262 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244256184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10304915791360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚206237317370880 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41575129 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚206237335373824 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚206237335373824 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚97899810 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚56324681 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚216542251165184 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10483269643776 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚196187821445120 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 244236118 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 147774922 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚56324683 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10483480215040 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚96481260 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚96481260 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚218514795807744 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 % _timestamp‚1.7884011866187131e+09 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 _runtime‚5811.256045033 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚216542064148992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10275203113176 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚228998276022784 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚206463041597656 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚196187838484480 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10304746778112 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 147774922 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚218514771867648 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚206237335373824 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚96481262 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 244256184 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10304915791360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚206237317370880 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26Ø•3uŽ2†- _wandbruntime‚5811  _runtime‚5811¼±ÞÙV‡Ð /loss/FBt2/detection_loss‚3.5034324377775192 #loss/total_loss‚44.700715417863 _runtime‚5811.257096588 /loss/FRt1/detection_loss‚2.2648458765311674 /loss/FRt2/detection_loss‚3.1247459691146324 .loss/FBt1/detection_loss‚2.592613642414411 1loss/SRqqt2/assignment_loss‚2.066753691062331 0loss/FBt1/assignment_loss‚0.6760581917866754 1loss/FBt2/assignment_loss‚0.49860907858121095 1loss/SRqqt2/detection_loss‚2.4164771251380444 0loss/SRqqt3/detection_loss‚1.861876175484874 0loss/FRt4/assignment_loss‚1.3562586581120726 0loss/SRqqt1/detection_loss‚1.766176554433843 /loss/FRt1/assignment_loss‚3.099870731884783 0loss/FRt3/assignment_loss‚2.6208948670206844 2loss/SRqqt1/assignment_loss‚1.3970729604046395 trainer/global_step‚7299 _step‚601 /loss/FRt2/assignment_loss‚4.116867945112031 1loss/SRqqt3/assignment_loss‚1.834970630028031 1loss/FBt4/assignment_loss‚0.18211283525237007 epoch‚6 .loss/FRt4/detection_loss‚1.497409103041301 1loss/SRqqt4/detection_loss‚1.0930992265192556 /loss/FBt4/detection_loss‚0.6919427020067814 2loss/SRqqt4/assignment_loss‚0.9315856759449546 0loss/FBt3/assignment_loss‚0.2030409848824723 /loss/FBt3/detection_loss‚2.4806183143095537 $ _timestamp‚1.788401186620356e+09 /loss/FRt3/detection_loss‚2.4233820410193623SŸF[ˆÕ 1loss/SRqqt2/assignment_loss‚2.066753691062331 0loss/FBt1/assignment_loss‚0.6760581917866754 1loss/FBt2/assignment_loss‚0.49860907858121095 1loss/SRqqt2/detection_loss‚2.4164771251380444 0loss/SRqqt3/detection_loss‚1.861876175484874 0loss/FRt4/assignment_loss‚1.3562586581120726 0loss/SRqqt1/detection_loss‚1.766176554433843 /loss/FRt1/assignment_loss‚3.099870731884783 0loss/FRt3/assignment_loss‚2.6208948670206844 2loss/SRqqt1/assignment_loss‚1.3970729604046395 trainer/global_step‚7299 _step‚601 /loss/FRt2/assignment_loss‚4.116867945112031 1loss/SRqqt3/assignment_loss‚1.834970630028031 1loss/FBt4/assignment_loss‚0.18211283525237007 epoch‚6 .loss/FRt4/detection_loss‚1.497409103041301 1loss/SRqqt4/detection_loss‚1.0930992265192556 /loss/FBt4/detection_loss‚0.6919427020067814 2loss/SRqqt4/assignment_loss‚0.9315856759449546 0loss/FBt3/assignment_loss‚0.2030409848824723 /loss/FBt3/detection_loss‚2.4806183143095537 $ _timestamp‚1.788401186620356e+09 /loss/FRt3/detection_loss‚2.4233820410193623 /loss/FBt2/detection_loss‚3.5034324377775192 #loss/total_loss‚44.700715417863 _runtime‚5811.257096588 /loss/FRt1/detection_loss‚2.2648458765311674 /loss/FRt2/detection_loss‚3.1247459691146324 .loss/FBt1/detection_loss‚2.592613642414411Ù¶VlI’‰‚(:‡ «¬ãÔ»…¦® gpu.0.gpu‚22.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚141.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.3634 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0a†sŠ‚(:è «¬ãÔ‰»½° disk.sda2.out‚0 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.percent‚16.4398193359375 proc.cpu.threads‚17 network.sent‚ 1.7843485e+07 disk.sda.out‚ 7651.03125 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0 proc.memory.rssMB‚2693.5( disk.nvme0c0n1.out‚28520.80029296875& disk.nvme0n1.out‚28520.80029296875$ disk./.usageGB‚229.5049705505371 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 378.94921875 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.0546875 disk.rbd1.out‚14.92822265625 disk.drbd2265.out‚0# disk.dm-0.out‚28520.80029296875$ memory_percent‚94.52874660491943( proc.memory.availableMB‚ 896.41015625 disk.dm-0.in‚ 378.94921875 disk.nvme0n1.in‚ 378.94921875 disk.drbd1716.in‚0 disk.dm-1.out‚ 7651.03125 network.recv‚ 2.079537e+06* disk./.usagePercent‚15.402271141143139 disk.sda.in‚ 0.0546875 disk.drbd2265.in‚0 disk.drbd2080.in‚0 cpu‚25.37913053535953 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda3.in‚ 0.0546875 disk.sda3.out‚ 7651.03125ŠÁ Í’‹‚(:‡ º¬ãÔþÊ ® gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚144.287' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.8574 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0U¥Äw|Œ‚(:ñ º¬ãÔÚæ›± disk.drbd2080.out‚0( proc.memory.availableMB‚ 1003.5234375 disk.sda1.out‚0 disk.sda2.in‚0 disk.drbd2265.out‚0$ memory_percent‚93.87497901916504 proc.memory.rssMB‚2693.5 disk.sda.out‚ 7651.48828125 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚28521.25341796875 disk.sda.in‚ 0.0546875 disk.drbd2265.in‚0 network.recv‚ 2.085007e+06 disk.sda2.out‚0* disk./.usagePercent‚15.402266020992453 disk.rbd1.out‚ 14.9912109375 disk.nvme0n1.in‚ 378.94921875 disk.dm-1.out‚ 7651.48828125 disk.drbd2080.in‚0 proc.cpu.threads‚17$ disk./.usageGB‚229.5048942565918 disk.sda3.out‚ 7651.48828125& disk.nvme0n1.out‚28521.25341796875 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.0546875( proc.memory.percent‚16.4398193359375 network.sent‚ 1.7951814e+07 disk.sda1.in‚0 disk.rbd1.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 378.94921875# disk.dm-0.out‚28521.25341796875 cpu‚25.378238695177437 disk.sda3.in‚ 0.0546875" disk.nvme0c0n1.in‚ 378.94921875 disk.drbd2266.in‚0¯iê2- _wandbruntime‚5846  _runtime‚5846ÙRÔÇŽÁ & lr-AdamW/pg1‚8.593107002602393e-05 & lr-AdamW/pg2‚8.593107002602393e-05 trainer/global_step‚7349 % _timestamp‚1.7884012216345088e+09 _runtime‚5846.271367142 _step‚602T¤…²ÌÆ _step‚602 & lr-AdamW/pg1‚8.593107002602393e-05 & lr-AdamW/pg2‚8.593107002602393e-05 trainer/global_step‚7349 % _timestamp‚1.7884012216345088e+09 _runtime‚5846.271367142ÚPAŒ 2- _wandbruntime‚5846  _runtime‚5846(ˆi¶2‘áO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚207573534245888 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10552721339392 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10373193962496 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10552934021120 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 trainer/global_step‚7349 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97101551 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚217946541205504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚207802340185776 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚207802158025012 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97101553 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10343285766832 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚537 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41840805 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚98527496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _step‚603 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚207573534245888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148747811 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚217946728208384 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10373024962560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10373024962560 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148727773 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚98526959 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚230484227670016 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10373193962496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56686156 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚98527496 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _step‚603 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚207573534245888 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 148747811 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚217946728208384 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10373024962560 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10373024962560 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 148727773 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚98526959 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚230484227670016 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10373193962496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚56686156 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚217946541205504 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚207802340185776 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚207802158025012 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚97101553 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10343285766832 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚537 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚41840805 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚207600803610624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 trainer/global_step‚7349 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10374587113984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98539580 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10374418100736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 245881836 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚207829672532864 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10344674931584 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97114210 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41846049 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚98540094 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _step‚604 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97114212 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 $ _timestamp‚1.788401222314922e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚207829490371892 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248u œj"•ïM M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚207829672532864 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10344674931584 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97114210 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚41846049 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚98540094 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _step‚604 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚97114212 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 $ _timestamp‚1.788401222314922e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚207829490371892 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚207600785607680 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 148747560 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224025600 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚219960443218944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 245861770 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97114212 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚207600803610624 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 trainer/global_step‚7349 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10374587113984 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚98539580 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10374418100736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 245881836 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 41071^­m¾Û2064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10623751810048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149720475 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚198756196496384 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10412592463836 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚219379680764928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚208936984479744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57055004 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚209168971120984 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97734501 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149740513 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10623539128320 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚219379867767808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚219379867767808 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99165930 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97734503 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10412757585240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚607 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚208936984479744 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10442696285184 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149740513´¾.áì'¤æO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚219379680764928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 149720475 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚198756213535744 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚7399 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚232000929491968 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚232000692870144 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚209168788960220 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 247475016 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚97734501 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 M@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚99165930 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚97734503 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10412757585240 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚607 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚208936984479744 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10442696285184 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 149740513 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚221377177681920 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 247454976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57055006 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10623751810048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 149720475 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚198756196496384 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10412592463836 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚219379680764928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚208936984479744 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57055004 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚209168971120984 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚97734501 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 149740513 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10623539128320 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚219379867767808 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚219379867767808 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10444258436608 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚99179199 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚232031027180544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚209196303468072 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚219408343267840 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42116816 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚221406066690048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10444089423360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @X _P:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149760326 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚208964253844480 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 trainer/global_step‚7399 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149740262 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚221406090630144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10625168964608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149740262 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149760326 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚208964271847424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10444258436608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57062383 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 % _timestamp‚1.7884012582836804e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97747160 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10624960490496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚198782139678720 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10413981628380 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 _step‚608 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚208964253844480 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10444089423360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42116277 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232031259594752 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99178658 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97747160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚209196121307100 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 247507488 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247507488 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚198782156718080 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 247487422 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚541 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096"m>¨ö&§ðM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42116277 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚232031259594752 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99178658 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚97747160 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚209196121307100 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 247507488 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 247507488 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚198782156718080 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 247487422 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚541 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 247487422 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10414146749992 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚97747162 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10444258436608 OBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚99179199 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚232031027180544 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚209196303468072 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚219408343267840 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42116816 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚221406066690048 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10444089423360 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 149760326 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚208964253844480 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 trainer/global_step‚7399 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 149740262 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚221406090630144 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10625168964608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 149740262 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 149760326 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚208964271847424 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10444258436608 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57062383 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 % _timestamp‚1.7884012582836804e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚97747160 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10624960490496 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚198782139678720 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10413981628380 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 _step‚608 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚208964253844480 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10444089423360à #ä 2¨- _wandbruntime‚5882  _runtime‚5882øŠ×8[©Õ epoch‚6 1loss/FBt3/assignment_loss‚0.11348618287593126 /loss/SRqqt3/detection_loss‚1.63525734895042 1loss/SRqqt4/detection_loss‚0.9378405626142218 /loss/FBt2/detection_loss‚2.5333494370182357 .loss/FBt3/detection_loss‚4.000600457191467 0loss/FRt1/assignment_loss‚3.7240759812149347 /loss/FRt2/assignment_loss‚3.841961834165785 /loss/FRt4/detection_loss‚1.5330530147254466 1loss/SRqqt1/detection_loss‚1.8887111719411154 1loss/SRqqt2/detection_loss‚2.3697487339377403 /loss/FRt4/assignment_loss‚1.354663234949112 2loss/SRqqt4/assignment_loss‚0.8362310363820964 0loss/FBt1/assignment_loss‚0.5873062024558229 /loss/FBt4/detection_loss‚0.7259892922441434 /loss/FRt1/detection_loss‚2.1194086238449694 /loss/FRt2/detection_loss‚3.0778859081091703 %loss/total_loss‚44.85441043563573 % _timestamp‚1.7884012582852833e+09 _runtime‚5882.922012096 2loss/SRqqt2/assignment_loss‚1.8933537991251796 1loss/FBt2/assignment_loss‚0.22748661142136703 /loss/FRt3/detection_loss‚2.5661304963601603 1loss/SRqqt3/assignment_loss‚1.662402896094136 1loss/FBt4/assignment_loss‚0.12060120427687925 /loss/FBt1/detection_loss‚2.5622928823743547 _step‚609 /loss/FRt3/assignment_loss‚2.744551993705131 2loss/SRqqt1/assignment_loss‚1.7980215296579094 trainer/global_step‚7399— ¦`ªÚ 1loss/SRqqt3/assignment_loss‚1.662402896094136 1loss/FBt4/assignment_loss‚0.12060120427687925 /loss/FBt1/detection_loss‚2.5622928823743547 _step‚609 /loss/FRt3/assignment_loss‚2.744551993705131 2loss/SRqqt1/assignment_loss‚1.7980215296579094 trainer/global_step‚7399 epoch‚6 1loss/FBt3/assignment_loss‚0.11348618287593126 /loss/SRqqt3/detection_loss‚1.63525734895042 1loss/SRqqt4/detection_loss‚0.9378405626142218 /loss/FBt2/detection_loss‚2.5333494370182357 .loss/FBt3/detection_loss‚4.000600457191467 0loss/FRt1/assignment_loss‚3.7240759812149347 /loss/FRt2/assignment_loss‚3.841961834165785 /loss/FRt4/detection_loss‚1.5330530147254466 1loss/SRqqt1/detection_loss‚1.8887111719411154 1loss/SRqqt2/detection_loss‚2.3697487339377403 /loss/FRt4/assignment_loss‚1.354663234949112 2loss/SRqqt4/assignment_loss‚0.8362310363820964 0loss/FBt1/assignment_loss‚0.5873062024558229 /loss/FBt4/detection_loss‚0.7259892922441434 /loss/FRt1/detection_loss‚2.1194086238449694 /loss/FRt2/detection_loss‚3.0778859081091703 %loss/total_loss‚44.85441043563573 % _timestamp‚1.7884012582852833e+09 _runtime‚5882.922012096 2loss/SRqqt2/assignment_loss‚1.8933537991251796 1loss/FBt2/assignment_loss‚0.22748661142136703 /loss/FRt3/detection_loss‚2.5661304963601603áW]™¯«‚(:’ ö¬ãÔ gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚125.402' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.080400000000004 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0žÚÌšs¬‚(:è ö¬ãÔÿ¬Ç± disk.sda2.in‚0$ memory_percent‚95.28610706329346( proc.memory.availableMB‚ 772.32421875 network.recv‚ 2.103693e+06! disk.dm-0.out‚28523.716796875 disk.dm-1.in‚ 0.0546875 network.sent‚ 1.8078077e+07 disk.sda2.out‚0 disk.dm-1.out‚ 7653.6953125 disk.sda1.out‚0 disk.sda3.out‚ 7653.6953125$ disk.nvme0n1.out‚28523.716796875 disk.drbd2266.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚229.5049057006836 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.0546875 disk.rbd1.in‚0 disk.rbd1.out‚ 14.9912109375" disk.nvme0c0n1.in‚ 378.94921875 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.drbd2266.in‚0( proc.memory.percent‚16.4398193359375 cpu‚25.374824553551722* disk./.usagePercent‚15.402266789015057 disk.rbd0.in‚0 disk.dm-0.in‚ 378.94921875& disk.nvme0c0n1.out‚28523.716796875 disk.sda.out‚ 7653.6953125 disk.drbd1716.in‚0 proc.memory.rssMB‚2693.5 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 378.94921875 disk.sda.in‚ 0.0546875ͤÐê­‚(:’ …­ãÔÁø„® gpu.0.gpu‚34.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚135.908' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.181599999999996 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀWðNc®‚(:Ø …­ãÔ¼öü° network.sent‚ 1.818676e+07* disk./.usagePercent‚15.402266789015057 disk.rbd1.in‚0 disk.rbd1.out‚ 14.9912109375 disk.sda.out‚7654.125& disk.nvme0c0n1.out‚28523.951171875' proc.memory.availableMB‚ 772.0859375 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.dm-1.out‚7654.125" disk.nvme0c0n1.in‚ 378.94921875 disk.sda.in‚ 0.0546875 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 378.94921875 cpu‚25.3740087439314 disk.drbd2265.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚229.5049057006836 disk.drbd2266.out‚0! disk.dm-0.out‚28523.951171875 proc.memory.rssMB‚2693.5 network.recv‚ 2.109874e+06 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.0546875 disk.nvme0n1.in‚ 378.94921875 disk.dm-1.in‚ 0.0546875 disk.sda3.out‚7654.125$ disk.nvme0n1.out‚28523.951171875$ memory_percent‚95.28756141662598( proc.memory.percent‚16.43981933593754ùn12¯- _wandbruntime‚5917  _runtime‚5917Ì9/Æ°À _step‚610 & lr-AdamW/pg1‚8.289998256962791e-05 & lr-AdamW/pg2‚8.289998256962791e-05 trainer/global_step‚7449 $ _timestamp‚1.788401293294176e+09 _runtime‚5917.931027401»NWO˱Š_runtime‚5917.931027401 _step‚610 & lr-AdamW/pg1‚8.289998256962791e-05 & lr-AdamW/pg2‚8.289998256962791e-05 trainer/global_step‚7449 $ _timestamp‚1.788401293294176e+09âËÑ×2²- _wandbruntime‚5917  _runtime‚5917RµìŒf$³àO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚210300470719488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99807724 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57423856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚5917.931862527 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150733215 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10512367607808 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42383331 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚233517228577280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57423854 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚233517471490560 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10694427424256 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10482064282244 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚220813007327232 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98367453 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚210535602056192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150733215 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚222822801153024 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚220813007327232 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚200053372652544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚220812820324352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10512536607744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98367451 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10694646397440 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 41071206JÃÚã€4 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10482229403648 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 $ _timestamp‚1.788401293294827e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚210535419895428 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚7449 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42383868 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚200053355613184 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217Ótó§ë'´åO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚220813007327232 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚98367453 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚210535602056192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 150733215 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚222822801153024 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚220813007327232 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚200053372652544 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚220812820324352 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10512536607744 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚98367451 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10694646397440 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10482229403648 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 $ _timestamp‚1.788401293294827e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚210535419895428 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚7449 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42383868 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚200053355613184 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚210300470719488 QDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚99807724 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚57423856 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 _runtime‚5917.931862527 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 150733215 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10512367607808 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42383331 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚233517228577280 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚57423854 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚233517471490560 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10694427424256 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10482064282244ã€}×!2µ- _wandbruntime‚5918  _runtime‚5918Šž=ñ&¶ëM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 249113074 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10513760745984 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚210327740084224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚210327740084224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 249113074 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210562752242308 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10513929759232 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚220841669843456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚222851714101248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57431231 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98380112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚200079315834880 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚222851738041344 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10696057275392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249133140 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚233547795316736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249133140 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150732964 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42389262 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99819958 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150753028 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 trainer/global_step‚7449 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10483618568400 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚5918.607972413 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚210327722081280 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10513929759232 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98380110 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150732964 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10513760745984 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚233547556611072 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚200079298795520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884012939706488e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98380110 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57431233 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚210562934403280 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98380112 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969a<Ùö&·ðM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 150753028 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42388727 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 249133140 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚233547795316736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 249133140 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 150732964 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42389262 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 K>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚99819958 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 150753028 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 trainer/global_step‚7449 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10483618568400 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚5918.607972413 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚210327722081280 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10513929759232 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚98380110 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 150732964 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10513760745984 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚233547556611072 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚200079298795520 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 % _timestamp‚1.7884012939706488e+09 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚98380110 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚57431233 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚210562934403280 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚98380112 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 249113074 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10513760745984 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚210327740084224 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚210327740084224 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 249113074 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚210562752242308 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10513929759232 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚220841669843456 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚222851714101248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57431231 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚98380112 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚200079315834880 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚222851738041344 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10696057275392 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚222245959883776 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚211663938956288 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚222246146886656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42654801 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99000401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100447507 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10582207930368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 trainer/global_step‚7499 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚224268448564224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚211902232991400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 250726320 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99000403 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10551536100652 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚201350514729984 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151705879 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42654241 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚222246146886656ÆÍ–í'ÄçO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚211663938956288 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 151725917 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚211663920953344 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218973184 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10582207930368 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10765507953664 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 PCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚99000403 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚222245959883776 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10582207930368 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 trainer/global_step‚7499 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚224268448564224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚211902232991400 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 250726320 TGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚99000403 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10551536100652 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚201350514729984 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 151705879 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42654241 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚222246146886656 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 100446945 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚235033737544704 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10765288980480 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚211902050830636 L?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚99000401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚201350531769344 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 250706280 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚222245959883776 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚211663938956288 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚222246146886656 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42654801 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99000401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 100447507 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100459758 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 % _timestamp‚1.7884013299011283e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250758792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99013062 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151745730 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚211929565338488 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚222274622386688 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235064066734592 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚201376457912320 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10766705222144 _runtime‚5954.538557049 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250738726 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10552925265196 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚211691208321024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10583601081856 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99013060 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚235064305440256 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57800081 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚211691190318080 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10583601081856 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚211691190318080 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99013060 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 250738726 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 trainer/global_step‚7499½4jø&ÇòM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚57800081 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚211691190318080 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10583601081856 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚211691190318080 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99013060 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 250738726 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 trainer/global_step‚7499 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚222274809402880 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 100459758 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 % _timestamp‚1.7884013299011283e+09 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 250758792 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99013062 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 151745730 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚211929565338488 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚222274622386688 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚235064066734592 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚201376457912320 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10766705222144 _runtime‚5954.538557049 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 250738726 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10552925265196 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚211691208321024 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10583601081856 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99013060 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚235064305440256 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199èŒ SŽ2È- _wandbruntime‚5954  _runtime‚5954âªÖöYÉÓ 0loss/FRt1/assignment_loss‚3.2345729023218155 0loss/FBt4/assignment_loss‚0.1501830272284811 0loss/SRqqt3/detection_loss‚1.951177621700547 trainer/global_step‚7499 /loss/FRt3/assignment_loss‚3.064110687971115 1loss/SRqqt4/detection_loss‚1.2658247480168938 /loss/FBt1/detection_loss‚2.7770914001898332 % _timestamp‚1.7884013299029138e+09 2loss/SRqqt2/assignment_loss‚1.6423539549352661 2loss/SRqqt3/assignment_loss‚1.4425844765522264 /loss/FRt2/detection_loss‚3.0145683650459563 /loss/FBt2/detection_loss‚3.1841404885053635 .loss/FBt4/detection_loss‚0.558912288377082 _step‚617 /loss/FRt2/assignment_loss‚3.523854753800801 0loss/FBt1/assignment_loss‚0.3787386185190991 0loss/FBt3/assignment_loss‚0.5745559824364516 /loss/FRt4/detection_loss‚1.8380132072738238 &loss/total_loss‚44.947029036232195 2loss/SRqqt1/assignment_loss‚1.5172862460603938 0loss/FBt2/assignment_loss‚0.3682136436837027 .loss/FRt3/detection_loss‚2.331001973748207 _runtime‚5954.539645793 /loss/FRt1/detection_loss‚2.5048449154604566 /loss/FBt3/detection_loss‚3.0784484148025513 0loss/FRt4/assignment_loss‚1.2160494114671434 1loss/SRqqt4/assignment_loss‚1.299705374489228 0loss/SRqqt1/detection_loss‚1.585395976435393 1loss/SRqqt2/detection_loss‚2.4454005572103683 epoch‚6¹ËÁï^ÊØ /loss/FRt4/detection_loss‚1.8380132072738238 &loss/total_loss‚44.947029036232195 2loss/SRqqt1/assignment_loss‚1.5172862460603938 0loss/FBt2/assignment_loss‚0.3682136436837027 .loss/FRt3/detection_loss‚2.331001973748207 _runtime‚5954.539645793 /loss/FRt1/detection_loss‚2.5048449154604566 /loss/FBt3/detection_loss‚3.0784484148025513 0loss/FRt4/assignment_loss‚1.2160494114671434 1loss/SRqqt4/assignment_loss‚1.299705374489228 0loss/SRqqt1/detection_loss‚1.585395976435393 1loss/SRqqt2/detection_loss‚2.4454005572103683 epoch‚6 0loss/FRt1/assignment_loss‚3.2345729023218155 0loss/FBt4/assignment_loss‚0.1501830272284811 0loss/SRqqt3/detection_loss‚1.951177621700547 trainer/global_step‚7499 /loss/FRt3/assignment_loss‚3.064110687971115 1loss/SRqqt4/detection_loss‚1.2658247480168938 /loss/FBt1/detection_loss‚2.7770914001898332 % _timestamp‚1.7884013299029138e+09 2loss/SRqqt2/assignment_loss‚1.6423539549352661 2loss/SRqqt3/assignment_loss‚1.4425844765522264 /loss/FRt2/detection_loss‚3.0145683650459563 /loss/FBt2/detection_loss‚3.1841404885053635 .loss/FBt4/detection_loss‚0.558912288377082 _step‚617 /loss/FRt2/assignment_loss‚3.523854753800801 0loss/FBt1/assignment_loss‚0.3787386185190991 0loss/FBt3/assignment_loss‚0.5745559824364516éô£½"’Ë‚(:‡ ²­ãÔµ¡‹® gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚126.172' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.2344 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0»qM-|Ì‚(:ñ ²­ãÔùÄ° disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0n1.in‚ 378.94921875 disk.dm-1.out‚ 7655.83203125 disk.drbd2266.in‚0 cpu‚25.371574342030097* disk./.usagePercent‚15.402266789015057 disk.rbd1.out‚ 14.9912109375 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17# disk.dm-0.out‚28525.35107421875& disk.nvme0n1.out‚28525.35107421875 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.0546875 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0$ memory_percent‚95.28639316558838 disk.sda3.out‚ 7655.83203125 disk.rbd1.in‚0( proc.memory.availableMB‚ 772.27734375 proc.memory.rssMB‚2693.5 disk.sda3.in‚ 0.0546875( disk.nvme0c0n1.out‚28525.35107421875 disk.sda.out‚ 7655.83203125 disk.sda2.out‚0( proc.memory.percent‚16.4398193359375 network.sent‚ 1.8307074e+07" disk.nvme0c0n1.in‚ 378.94921875 disk.sda.in‚ 0.0546875 disk.drbd2265.out‚0$ disk./.usageGB‚229.5049057006836 disk.dm-0.in‚ 378.94921875 network.recv‚ 2.124115e+06 disk.drbd1716.out‚0 disk.drbd2080.in‚0Æø&3œÍ‚(:‘ Á­ãÔ€»÷­ gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚128.89' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.777999999999995 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0F/àºs΂(:è Á­ãÔ°¢ô° disk.sda3.out‚ 7656.296875& disk.nvme0n1.out‚28525.60498046875( proc.memory.percent‚16.4398193359375 disk.dm-1.out‚ 7656.296875 disk.drbd2266.in‚0# disk.dm-0.out‚28525.60498046875 disk.sda1.in‚0 network.recv‚ 2.129891e+06 disk.drbd2266.out‚0* disk./.usagePercent‚15.402266789015057 disk.nvme0n1.in‚ 378.94921875 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.sda1.out‚0% proc.memory.availableMB‚ 772.46875 proc.memory.rssMB‚2693.5 cpu‚25.370766591633526 network.sent‚ 1.8414778e+07 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0$ memory_percent‚95.28522491455078 disk.rbd1.in‚0 disk.sda.in‚ 0.0546875 disk.sda.out‚ 7656.296875 disk.dm-1.in‚ 0.0546875 disk.dm-0.in‚ 378.94921875$ disk./.usageGB‚229.5049057006836 disk.sda3.in‚ 0.0546875" disk.nvme0c0n1.in‚ 378.94921875 disk.drbd1716.in‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 14.9912109375( disk.nvme0c0n1.out‚28525.60498046875hï|â’Ï‚(:‡ ЭãÔÂñ”® gpu.0.gpu‚34.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚127.528' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.5056 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0FåqºuЂ(:ê ЭãÔ¶–ö± disk.rbd0.in‚0 disk.rbd1.out‚ 14.9912109375 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.dm-1.out‚ 7657.23828125 disk.sda.out‚ 7657.23828125 disk.rbd0.out‚0 disk.dm-0.in‚ 378.94921875' proc.memory.availableMB‚ 772.2109375 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 7657.23828125 disk.nvme0n1.in‚ 378.94921875 network.sent‚ 1.8420673e+07* disk./.usagePercent‚15.402266789015057 disk.dm-0.out‚28526.23046875 disk.sda3.in‚ 0.05859375 disk.rbd1.in‚0 disk.sda2.out‚0$ memory_percent‚95.28679847717285( proc.memory.percent‚16.4398193359375 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0# disk.nvme0n1.out‚28526.23046875 disk.sda2.in‚0 proc.memory.rssMB‚2693.5 network.recv‚ 2.133873e+06 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 378.94921875$ disk./.usageGB‚229.5049057006836 disk.dm-1.in‚ 0.05859375 disk.drbd2266.in‚0% disk.nvme0c0n1.out‚28526.23046875 disk.sda.in‚ 0.05859375 cpu‚25.369957011904273ò4Ìí2Ñ- _wandbruntime‚5989  _runtime‚5989ʹõOÇÒÁ % _timestamp‚1.7884013648505478e+09 _runtime‚5989.487378901 _step‚618 & lr-AdamW/pg1‚7.988939489279513e-05 & lr-AdamW/pg2‚7.988939489279513e-05 trainer/global_step‚7549&N—ÌÓÆ & lr-AdamW/pg2‚7.988939489279513e-05 trainer/global_step‚7549 % _timestamp‚1.7884013648505478e+09 _runtime‚5989.487378901 _step‚618 & lr-AdamW/pg1‚7.988939489279513e-05ên­g22Ô- _wandbruntime‚5989  _runtime‚5989¼Êªè'ÕâO PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99633351 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 152698581 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226136064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚223679099443200 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚225714095975424 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚213027407193088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚213268681765844 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42926736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚225714119915520 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10651879252992 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152718619 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 152698581 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10836151866880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10621007919060 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202195968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58161556 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚202647690886144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚202647673846784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 252331932 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚236550473978368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚213268863926608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10651879252992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42927236 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 $ _timestamp‚1.788401364851228e+09 _runtime‚5989.488260374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚223679286446080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10651710253056 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 252351972 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58161554 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199jFéðÖçO G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚42926736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚225714119915520 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10651879252992 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 152718619 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 152698581 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10836151866880 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10621007919060 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202195968 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58161556 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚202647690886144 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚202647673846784 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 2523b}i^ 31932 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚236550473978368 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚213268863926608 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10651879252992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚42927236 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 $ _timestamp‚1.788401364851228e+09 _runtime‚5989.488260374 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚223679286446080 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10651710253056 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 252351972 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58161554 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 PCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚99633351 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 152698581 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226136064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚223679099443200 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚225714095975424 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚213027407193088 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚213268681765844ë•=_2×- _wandbruntime‚5990  _runtime‚5990ˆe‹ò&ØìM $ _timestamp‚1.788401365521239e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99646012 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58168931 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10653272404480 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚488 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚223707761946112 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚223707948962304 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚213054658554880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10837780782080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10837563919360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 152718368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 252364378 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58168933 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 252384444 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42932655 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10653272404480 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 152738432 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚225743008923648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99646010 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚223707948962304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10653103391232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 152718368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚7549 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 101101588 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101101098 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚213296196273696 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚213054676557824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99646012 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _runtime‚5990.158667045 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 252364378 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚213054676557824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42932167 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10622562205216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚490 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10622397083604 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10653103391232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚236580572843008 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚236580813645824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99646010 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 _step‚620 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252384444›l¤›÷&ÙñM ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10653272404480 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 152738432 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚225743008923648 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 NADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚99646010 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚223707948962304 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10653103391232 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 152718368 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 trainer/global_step‚7549 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 101101588 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101101098 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚213296196273696 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚213054676557824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 NADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚99646012 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _runtime‚5990.158667045 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 252364378 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚213054676557824 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚42932167 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚10622562205216 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚490 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10622397083604 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10653103391232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚236580572843008 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚236580813645824 J=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚99646010 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 _step‚620 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 252384444 $ _timestamp‚1.788401365521239e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚99646012 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58168931 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10653272404480 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚488 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚223707761946112 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚223707948962304 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚213054658554880 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10837780782080 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10837563919360 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 152718368 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 252364378 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58168933 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 252384444 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚42932655 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0ìÉ2Ú- _wandbruntime‚5990  _runtime‚5990qA†Å]Û× /loss/FBt4/detection_loss‚0.5740108044527761 1loss/SRqqt3/detection_loss‚1.5517275584240755 .loss/FBt1/detection_loss‚2.952099016734532 /loss/FBt3/detection_loss‚1.8932545615567102 2loss/SRqqt2/assignment_loss‚2.1633928252117975 2loss/SRqqt4/assignment_loss‚1.1486539011347596 /loss/FRt1/detection_loss‚2.1976429665088655 0loss/FRt2/assignment_loss‚3.7382148432009146 0loss/FBt3/assignment_loss‚0.3762413842551824 1loss/SRqqt1/detection_loss‚1.5750386518127513 epoch‚6 % _timestamp‚1.7884013655231912e+09 /loss/FRt2/detection_loss‚2.8853126240499094 /loss/FRt4/detection_loss‚1.6008239521692365 &loss/total_loss‚44.498374625029975 0loss/FRt4/assignment_loss‚1.4838160387540267 .loss/FRt3/detection_loss‚2.310037623454885 _runtime‚5990.159981756 /loss/FRt1/assignment_loss‚3.868466416597366 1loss/FBt1/assignment_loss‚0.16930000462285275 1loss/FBt4/assignment_loss‚0.20767641613697424 trainer/global_step‚7549 _step‚621 /loss/FRt3/assignment_loss‚2.777380556595035 2loss/SRqqt1/assignment_loss‚1.9010247213168527 2loss/SRqqt3/assignment_loss‚1.6703253127013644 1loss/FBt2/assignment_loss‚0.24969111708818068 1loss/SRqqt2/detection_loss‚2.2981494196823666 0loss/SRqqt4/detection_loss‚1.302629632970034 .loss/FBt2/detection_loss‚3.603464275598526¨XÝõbÜÜ 0loss/SRqqt4/detection_loss‚1.302629632970034 .loss/FBt2/detection_loss‚3.603464275598526 /loss/FBt4/detection_loss‚0.5740108044527761 1loss/SRqqt3/detection_loss‚1.5517275584240755 .loss/FBt1/detection_loss‚2.952099016734532 /loss/FBt3/detection_loss‚1.8932545615567102 2loss/SRqqt2/assignment_loss‚2.1633928252117975 2loss/SRqqt4/assignment_loss‚1.1486539011347596 /loss/FRt1/detection_loss‚2.1976429665088655 0loss/FRt2/assignment_loss‚3.7382148432009146 0loss/FBt3/assignment_loss‚0.3762413842551824 1loss/SRqqt1/detection_loss‚1.5750386518127513 epoch‚6 % _timestamp‚1.7884013655231912e+09 /loss/FRt2/detection_loss‚2.8853126240499094 /loss/FRt4/detection_loss‚1.6008239521692365 &loss/total_loss‚44.498374625029975 0loss/FRt4/assignment_loss‚1.4838160387540267 .loss/FRt3/detection_loss‚2.310037623454885 _runtime‚5990.159981756 /loss/FRt1/assignment_loss‚3.868466416597366 1loss/FBt1/assignment_loss‚0.16930000462285275 1loss/FBt4/assignment_loss‚0.20767641613697424 trainer/global_step‚7549 _step‚621 /loss/FRt3/assignment_loss‚2.777380556595035 2loss/SRqqt1/assignment_loss‚1.9010247213168527 2loss/SRqqt3/assignment_loss‚1.6703253127013644 1loss/FBt2/assignment_loss‚0.24969111708818068 1loss/SRqqt2/detection_loss‚2.2981494196823666í (·Ý‚(:’ ß­ãÔܱ’® gpu.0.gpu‚35.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚133.733' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.746599999999997 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0â–°¤yÞ‚(:î ß­ãÔ½”‚±* disk./.usagePercent‚15.402269861105466 disk.drbd2080.in‚0 disk.rbd1.in‚0# memory_percent‚95.2854871749878 proc.memory.rssMB‚2693.5 disk.sda.in‚ 0.05859375 disk.sda.out‚ 7657.75390625( proc.memory.percent‚16.4398193359375 disk.drbd2266.out‚0 disk.drbd2080.out‚0! disk.dm-0.out‚28526.529296875$ disk.nvme0n1.out‚28526.529296875 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 7657.75390625 network.sent‚ 1.8427019e+07 disk.rbd0.in‚0 disk.dm-0.in‚ 378.94921875 disk.rbd1.out‚ 14.9912109375" disk.nvme0c0n1.in‚ 378.94921875 disk.nvme0n1.in‚ 378.94921875 disk.dm-1.out‚ 7657.75390625( proc.memory.availableMB‚ 772.42578125 network.recv‚ 2.138449e+06 disk.rbd0.out‚0 disk.sda3.in‚ 0.05859375& disk.nvme0c0n1.out‚28526.529296875 disk.sda2.in‚0 disk.dm-1.in‚ 0.05859375 cpu‚25.369168320139657 proc.cpu.threads‚17% disk./.usageGB‚229.50495147705078 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0JrÏø:ß‚(:’ î­ãÔ«‚® gpu.0.gpu‚42.0 gpu.0.mem©IéHcory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚125.402' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.080400000000004 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0DŒ¬?zà‚(:ï î­ãÔºÐÙ± network.recv‚ 2.143747e+06 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0# disk.dm-0.out‚28540.37744140625 disk.sda1.out‚0$ memory_percent‚95.28701305389404 proc.memory.rssMB‚2693.5) disk./.usagePercent‚15.40305734028079 disk.drbd2265.in‚0( proc.memory.percent‚16.4398193359375 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 378.94921875( disk.nvme0c0n1.out‚28540.37744140625 disk.sda.out‚ 7658.3671875 network.sent‚ 1.853496e+07 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.sda3.in‚ 0.05859375 disk.sda3.out‚ 7658.3671875( proc.memory.availableMB‚ 772.17578125% disk./.usageGB‚229.51668548583984 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 378.94921875 disk.sda2.in‚0 disk.dm-1.in‚ 0.05859375 disk.dm-1.out‚ 7658.3671875 disk.sda1.in‚0 disk.rbd1.out‚ 14.9912109375 disk.sda.in‚ 0.05859375 cpu‚25.36836978314175 disk.dm-0.in‚ 378.94921875 disk.rbd1.in‚0& disk.nvme0n1.out‚28540.37744140625 proc.cpu.threads‚17é(λ2á- _wandbruntime‚6025  _runtime‚60259æOÇâÁ & lr-AdamW/pg1‚7.839257340809162e-05 & lr-AdamW/pg2‚7.839257340809162e-05 trainer/global_step‚7599 % _timestamp‚1.7884014007075984e+09 _runtime‚6025.344466095 _step‚622°»DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 253957584 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43197872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10907244012032 trainer/global_step‚7599 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚225112239002624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚227159743386624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚214635312701052 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100266303 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153691283 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚225112426005504 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚225112426005504 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚550 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100266303 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 101727726 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚203944850002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153711321 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 153691283 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10721381575680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚214390857426944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚227159767326720\+–úò'æìO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚214635494861816 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10721381575680 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚214390875429888 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10690644858872 _runtime‚6025.345530063 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 100266301 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 153711321 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚238067011338752 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 253957584 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43197872 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10907244012032 trainer/global_step‚7599 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚225112239002624 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚552 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚227159743386624 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚214635312701052 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 100266303 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 153691283 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚225112426005504 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚225112426005504 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚550 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100266303 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 101727726 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚203944850002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 153711321 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 153691283 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10721381575680 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚214390857426944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚227159767326720 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100266301 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 253977624 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153711070 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58537783 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 _runtime‚6026.017112366 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 253990030 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10722943727104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 $ _timestamp‚1.788401401379715e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚225140901505536 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚225141088521728 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100278962 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153731134 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚238097338061312 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚203970776145920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚214418126791680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43203232 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10908657786368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚214418144794624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100278960 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚227188680274944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚238097105647104 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10908449312256 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10722774713856 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101740502 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚214662827208904 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚214662645047932 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚225141088521728 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100278960 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153731134 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10722943727104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚214418126791680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚7599 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43202721 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254010096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10691868902012 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58537781 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153711070 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚203970793185280 _step‚624¶õ±ØéõM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚203970776145920 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚214418126791680 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43203232 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10908657786368 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚214418144794624 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100278960 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚227188680274944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚238097105647104 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10908449312256 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10722774713856 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 101740502 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚214662827208904 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚214662645047932 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚225141088521728 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100278960 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 153731134 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10722943727104 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚214418126791680 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚7599 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43202721 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 254010096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10691868902012 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58537781 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 153711070 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚203970793185280 _step‚624 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 153711070 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58537783 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 _runtime‚6026.017112366 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 253990030 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10722943727104 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 $ _timestamp‚1.788401401379715e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚225140901505536 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚225141088521728 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100278962 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 153731134 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚238097338061312 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248ðó.f2ê- _wandbruntime‚6026  _runtime‚6026\Þ÷EWëÑ /loss/FRt4/detection_loss‚1.9227345497125672 trainer/global_step‚7599 % _timestamp‚1.7884014013815298e+09 -loss/FRt3/assignment_loss‚3.4395621500232 0loss/FBt1/assignment_loss‚0.3882453613979963 0loss/FBt3/assignment_loss‚0.4179942044928187 1loss/SRqqt2/detection_loss‚2.8030656364656266 0loss/SRqqt3/detection_loss‚1.863500042706728 .loss/FBt1/detection_loss‚2.292278092641097 0loss/FRt4/assignment_loss‚1.5948280621142614 2loss/SRqqt2/assignment_loss‚1.7675931756534884 1loss/FBt4/assignment_loss‚0.14284211238907302 1loss/SRqqt1/detection_loss‚1.8788751022382215 1loss/SRqqt4/detection_loss‚1.0432818047110997 /loss/FBt2/detection_loss‚2.2096398898533414 /loss/FBt3/detection_loss‚2.9973346259858875 _step‚625 2loss/SRqqt4/assignment_loss‚0.9417389506208045 0loss/FBt2/assignment_loss‚0.3419524518364402 -loss/FRt1/detection_loss‚2.10913535192901 /loss/FBt4/detection_loss‚0.5121605973912492 /loss/FRt2/assignment_loss‚4.352777446017546 .loss/FRt2/detection_loss‚2.741025786189472 _runtime‚6026.018308172 0loss/FRt1/assignment_loss‚3.0995926588189366 epoch‚6 /loss/FRt3/detection_loss‚2.5965908364477484 2loss/SRqqt1/assignment_loss‚1.4422163368626075 1loss/SRqqt3/assignment_loss‚1.083540382757783 &loss/total_loss‚43.982505609257004üi¥›\ìÖ /loss/FRt3/detection_loss‚2.5965908364477484 2loss/SRqqt1/assignment_loss‚1.4422163368626075 1loss/SRqqt3/assignment_loss‚1.083540382757783 &loss/total_loss‚43.982505609257004 /loss/FRt4/detection_loss‚1.9227345497125672 trainer/global_step‚7599 % _timestamp‚1.7884014013815298e+09 -loss/FRt3/assignment_loss‚3.4395621500232 0loss/FBt1/assignment_loss‚0.3882453613979963 0loss/FBt3/assignment_loss‚0.4179942044928187 1loss/SRqqt2/detection_loss‚2.8030656364656266 0loss/SRqqt3/detection_loss‚1.863500042706728 .loss/FBt1/detection_loss‚2.292278092641097 0loss/FRt4/assignment_loss‚1.5948280621142614 2loss/SRqqt2/assignment_loss‚1.7675931756534884 1loss/FBt4/assignment_loss‚0.14284211238907302 1loss/SRqqt1/detection_loss‚1.8788751022382215 1loss/SRqqt4/detection_loss‚1.0432818047110997 /loss/FBt2/detection_loss‚2.2096398898533414 /loss/FBt3/detection_loss‚2.9973346259858875 _step‚625 2loss/SRqqt4/assignment_loss‚0.9417389506208045 0loss/FBt2/assignment_loss‚0.3419524518364402 -loss/FRt1/detection_loss‚2.10913535192901 /loss/FBt4/detection_loss‚0.5121605973912492 /loss/FRt2/assignment_loss‚4.352777446017546 .loss/FRt2/detection_loss‚2.741025786189472 _runtime‚6026.018308172 0loss/FRt1/assignment_loss‚3.0995926588189366 epoch‚6ñJ(Øí‚(:’ ý­ãÔɺ¥® gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚117.311' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.462200000000003 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0i´Ñtî‚(:é ý­ãÔ÷ î° disk.rbd1.out‚ 14.9912109375" disk.nvme0c0n1.in‚ 378.94921875 disk.dm-1.in‚ 0.05859375 disk.sda3.in‚ 0.05859375 disk.sda2.in‚0 disk.drbd2080.out‚0) disk./.usagePercent‚15.40305734028079 disk.dm-0.in‚ 378.94921875# disk.dm-0.out‚28540.63134765625 disk.drbd2080.in‚0 network.sent‚ 1.854103e+07 disk.sda.out‚ 7658.65625 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 378.94921875 disk.sda.in‚ 0.05859375 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚28540.63134765625 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚0 proc.memory.rssMB‚2693.5 cpu‚25.367547002479878 disk.dm-1.out‚ 7658.65625 proc.cpu.threads‚17 network.recv‚ 2.147974e+06 disk.sda1.in‚0 disk.sda3.out‚ 7658.65625& disk.nvme0n1.out‚28540.63134765625# memory_percent‚95.2843427658081( proc.memory.availableMB‚ 772.61328125( proc.memory.percent‚16.4398193359375% disk./.usageGB‚229.51668548583984 disk.rbd1.in‚0hF²~’ï‚(:‡ Œ®ãÔɆ§® gpu.0.gpu‚32.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚127.528' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.5056 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏäOÓoð‚(:ä Œ®ãỔþ± disk.drbd2266.out‚0 disk.dm-0.in‚ 378.94921875 cpu‚25.3667526515422 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.05859375 disk.nvme0n1.in‚ 378.94921875 disk.sda2.out‚0 disk.sda1.out‚0 disk.sda.out‚ 7659.3125 disk.sda3.in‚ 0.05859375$ memory_percent‚95.28734683990479 network.sent‚ 1.8649749e+07* disk./.usagePercent‚15.403058876325995 disk.sda.in‚ 0.05859375 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0( proc.memory.percent‚16.4398193359375 proc.cpu.threads‚17 network.recv‚ 2.153942e+06 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.sda3.out‚ 7659.3125 disk.rbd1.out‚ 14.9912109375 disk.dm-1.out‚ 7659.3125% disk.nvme0n1.out‚28541.1630859375 disk.sda2.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 378.94921875( proc.memory.availableMB‚ 772.12109375 disk.drbd2080.out‚0 disk.rbd0.in‚0" disk.dm-0.out‚28541.1630859375 disk.rbd1.in‚0' disk.nvme0c0n1.out‚28541.1630859375 proc.memory.rssMB‚2693.5% disk./.usageGB‚229.51670837402344䫃H›ñ‚(: ›®ãÔ¯˜Ÿ® gpu.0.gpu‚36.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚137.8' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.560000000000002 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º“±sò‚(:è ›®ãÔ³¢é° disk.drbd2080.in‚0 disk.dm-0.in‚ 378.94921875& proc.memory.availableMB‚ 772.109375 network.sent‚ 1.8656058e+07 disk.rbd1.out‚ 14.9912109375 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.drbd2265.out‚0) disk./.usagePercent‚15.40306271643901' disk.nvme0c0n1.out‚28541.3779296875 disk.nvme0n1.in‚ 378.94921875 disk.sda.out‚ 7659.796875 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.sda3.in‚ 0.05859375 disk.sda3.out‚ 7659.796875 disk.sda2.in‚0 disk.drbd2266.out‚0 cpu‚25.365981812046034% disk./.usageGB‚229.51676559448242 disk.sda1.out‚0% disk.nvme0n1.out‚28541.3779296875 disk.drbd2266.in‚0 disk.rbd0.in‚0 proc.memory.rssMB‚2693.5 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 378.94921875 disk.sda1.in‚0 disk.sda.in‚ 0.05859375 disk.rbd0.out‚0( proc.memory.percent‚16.4398193359375 network.recv‚ 2.15839e+06 disk.dm-1.in‚ 0.05859375 disk.dm-1.out‚ 7659.796875" disk.dm-0.out‚28541.3779296875$ memory_percent‚95.28741836547852 disk.rbd1.in‚0 B¥Ù2ó- _wandbruntime‚6061  _runtime‚6061ÿí<³ÇôÁ % _timestamp‚1.7884014363729706e+09 _runtime‚6061.009824281 _step‚626 & lr-AdamW/pg1‚7.690181647261388e-05 & lr-AdamW/pg2‚7.690181647261388e-05 trainer/global_step‚7649=u%ÌõÆ trainer/global_step‚7649 % _timestamp‚1.7884014363729706e+09 _runtime‚6061.009824281 _step‚626 & lr-AdamW/pg1‚7.690181647261388e-05 & lr-AdamW/pg2‚7.690181647261388e-05òÛ2ö- _wandbruntime‚6061  _runtime‚6061ýÐI;í'÷çO LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10791052898304 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10791052898304 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10760116677280 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 255583236 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚215754343666688 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10759951555876 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10977950423040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100899251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10791221898240 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100899253 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102366358 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚205241992080384 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚215754325663744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58899254 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚239583579939840 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚627 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154704023 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚7649 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚226545565564928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154683985 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43467102 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚226545565564928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚226545378562048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43466606 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚205242009119744 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚239583341220864 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884014363736694e+09 _runtime‚6061.010754636 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚498 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚215754325663744 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚216001943636260 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10978165201920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚228605390797824 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚226545378562048 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10791221898240 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 255603276 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154683985 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102365860 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58899256 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 154704023÷Ç{Q„øìO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚226545565564928 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚226545378562048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43466606 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚205242009119744 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚239583341220864 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 % _timestamp‚1.7884014363736694e+09 _runtime‚6061.010754636 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚498 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚215754325663744 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚216001943636260 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚10978165201920 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚228605390797824 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚226545378562048 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10791221898240 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 255603276 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 154683985 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 46#n10712064 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 102365860 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚58899256 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 154704023 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10791052898304 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10791052898304 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10760116677280 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 255583236 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚215754343666688 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10759951555876 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚10977950423040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 100899251 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚10791221898240 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 100899253 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 102366358 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚205241992080384 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚215754325663744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚58899254 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚239583579939840 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚627 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 154704023 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 trainer/global_step‚7649 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚226545565564928 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 154683985 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43467102 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0óuwƒ½2ù- _wandbruntime‚6061  _runtime‚6061.Ü”`ö&úðM ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10979375371776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚628 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58906631 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102378422 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154723836 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 $ _timestamp‚1.788401437043267e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 255635748 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100911912 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚215781595028480 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58906633 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43472310 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚216029275983140 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100911912 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚226574228081152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215781595028480 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚205267952302080 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 255615682 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7649 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10761340720420 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚6061.685799447 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100911910 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10792615049728 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10792615049728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 255615682 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154723836 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚205267935262720 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100911910 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154703772 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚239613919920640 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255635748 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43471791 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚239613679117824 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚228634303746048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚216029458144112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚226574228081152 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102378943 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10792446036480 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10979592234496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚215781613031424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154703772 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚228634327686144 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416àrBÍû&ûõM O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 255635748 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 100911912 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚215781595028480 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚58906633 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43472310 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚216029275983140 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 100911912 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚226574228081152 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚215781595028480 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚205267952302080 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 255615682 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚521 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚7649 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10761340720420 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚6061.685799447 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 100911910 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10792615049728 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10792615049728 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 255615682 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 154723836 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚205267935262720 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 100911910 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 154703772 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚239613919920640 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 255635748 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43471791 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚239613679117824 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚228634303746048 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚216029458144112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚226574228081152 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 102378943 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10792446036480 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚10979592234496 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚215781613031424 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 154703772 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚228634327686144 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚10979375371776 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚628 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚58906631 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 102378422 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 154723836 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 $ _timestamp‚1.788401437043267e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192ôéî(b2ü- _wandbruntime‚6061  _runtime‚6061!ÉXýÒ 0loss/FRt3/assignment_loss‚2.5416712250028337 .loss/FRt2/detection_loss‚2.797113434834914 _step‚629 1loss/SRqqt2/assignment_loss‚2.473607468108336 0loss/SRqqt3/detection_loss‚1.851796083773176 1loss/SRqqt4/detection_loss‚1.0481388315868874 trainer/global_step‚7649 /loss/FRt1/assignment_loss‚2.789607686025125 0loss/FRt2/assignment_loss‚3.4441766955635766 /loss/FBt4/detection_loss‚0.6882272031748804 0loss/SRqqt1/assignment_loss‚1.70889770524933 1loss/FBt4/assignment_loss‚0.15825980008317472 0loss/SRqqt2/detection_loss‚2.469186711642477 &loss/total_loss‚42.842799093075186 epoch‚6 0loss/FRt4/assignment_loss‚1.6546608593504308 1loss/SRqqt3/assignment_loss‚1.575597445325305 /loss/FRt3/detection_loss‚2.5979415125080516 1loss/SRqqt1/detection_loss‚1.8626783559931086 % _timestamp‚1.7884014370501628e+09 0loss/FBt1/assignment_loss‚0.4008446778573102 1loss/FBt3/assignment_loss‚0.35102796934855246 .loss/FRt4/detection_loss‚1.651099478226092 .loss/FBt2/detection_loss‚2.701782767261778 1loss/FBt2/assignment_loss‚0.32615152599932323 /loss/FBt1/detection_loss‚2.2316887617111205 2loss/SRqqt4/assignment_loss‚1.1995793474165515 /loss/FRt1/detection_loss‚2.1792448538321034 .loss/FBt3/detection_loss‚2.139818693200747 _runtime‚6061.686916445רäà]þ× 1loss/SRqqt2/assignment_loss‚2.473607468108336 0loss/SRqqt3/detection_loss‚1.851796083773176 1loss/SRqqt4/detection_loss‚1.0481388315868874 trainer/global_step‚7649 /loss/FRt1/assignment_loss‚2.789607686025125 0loss/FRt2/assignment_loss‚3.4441766955635766 /loss/FBt4/detection_loss‚0.6882272031748804 0loss/SRqqt1/assignment_loss‚1.70889770524933 1loss/FBt4/assignment_loss‚0.15825980008317472 0loss/SRqqt2/detection_loss‚2.469186711642477 &loss/total_loss‚42.842799093075186 epoch‚6 0loss/FRt4/assignment_loss‚1.6546608593504308 1loss/SRqqt3/assignment_loss‚1.575597445325305 /loss/FRt3/detection_loss‚2.5979415125080516 1loss/SRqqt1/detection_loss‚1.8626783559931086 % _timestamp‚1.7884014370501628e+09 0loss/FBt1/assignment_loss‚0.4008446778573102 1loss/FBt3/assignment_loss‚0.35102796934855246 .loss/FRt4/detection_loss‚1.651099478226092 .loss/FBt2/detection_loss‚2.701782767261778 1loss/FBt2/assignment_loss‚0.32615152599932323 /loss/FBt1/detection_loss‚2.2316887617111205 2loss/SRqqt4/assignment_loss‚1.1995793474165515 /loss/FRt1/detection_loss‚2.1792448538321034 .loss/FBt3/detection_loss‚2.139818693200747 _runtime‚6061.686916445 0loss/FRt3/assignment_loss‚2.5416712250028337 .loss/FRt2/detection_loss‚2.797113434834914 _step‚629õZhEL’ÿ‚(:‡ ª®ãԿϲ® gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚120.262' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0524 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0%#|€‚(:ñ ª®ãÔÀë±$ disk./.usageGB‚229.5166893005371 disk.nvme0n1.in‚ 378.94921875 disk.rbd0.out‚0 disk.dm-0.out‚28542.39453125* proc.memory.percent‚15.888381004333496 disk.rbd1.out‚ 14.9912109375 disk.sda.out‚ 7660.55078125 disk.drbd1716.in‚0 disk.dm-1.out‚ 7660.55078125 network.sent‚ 1.8661373e+07 network.recv‚ 2.161815e+06* disk./.usagePercent‚15.403057596288324 disk.sda1.in‚0$ memory_percent‚94.74430084228516 cpu‚25.365196109404486 disk.sda3.out‚ 7660.55078125 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.sda3.in‚ 0.05859375 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.05859375 disk.drbd2266.in‚0% disk.nvme0c0n1.out‚28542.39453125# disk.nvme0n1.out‚28542.39453125 disk.drbd2266.out‚0 disk.dm-0.in‚ 378.94921875 disk.sda1.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 378.94921875 disk.drbd2265.out‚0 disk.sda.in‚ 0.05859375 disk.sda2.out‚0 disk.drbd2265.in‚0% proc.memory.availableMB‚ 861.09375# proc.memory.rssMB‚ 2603.15234375 proc.cpu.threads‚17Bû ‚(:’ ¹®ãÔ䈎® gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚117.841' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.568199999999997 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ú‹Æ}‚‚(:ò ¹®ãÔù¦þ°% disk./.usageGB‚229.51670837402344 disk.sda.in‚ 0.05859375 disk.sda.out‚ 7661.0078125 proc.cpu.threads‚17 disk.rbd0.in‚0 disk.sda1.in‚0 disk.dm-1.out‚ 7661.0078125 disk.drbd2266.out‚0 disk.drbd2265.out‚0* disk./.usagePercent‚15.403058876325995 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 378.94921875 disk.dm-0.out‚28542.75390625 disk.sda3.out‚ 7661.0078125 disk.drbd2265.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.05859375 disk.rbd1.in‚0 disk.rbd1.out‚ 14.9912109375 disk.sda2.in‚0( proc.memory.availableMB‚ 860.55078125 cpu‚25.364389285600993 network.recv‚ 2.168259e+06 disk.drbd2080.in‚0% disk.nvme0c0n1.out‚28542.75390625 disk.nvme0n1.in‚ 378.94921875 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 378.94921875# disk.nvme0n1.out‚28542.75390625 disk.dm-1.in‚ 0.05859375$ memory_percent‚94.74761486053467# proc.memory.rssMB‚ 2603.15234375* proc.memory.percent‚15.888381004333496 network.sent‚ 1.8770206e+07ðd-2ƒ- _wandbruntime‚6096  _runtime‚6096€±ç€Ã„½ _runtime‚6096.829986531 _step‚630 $ lr-AdamW/pg1‚7.5417434757873e-05 $ lr-AdamW/pg2‚7.5417434757873e-05 trainer/global_step‚7699 % _timestamp‚1.7884014721931396e+09g ñÈ…Â trainer/global_step‚7699 % _timestamp‚1.7884014721931396e+09 _runtime‚6096.829986531 _step‚630 $ lr-AdamW/pg1‚7.5417434757873e-05 $ lr-AdamW/pg2‚7.5417434757873e-05ölŽÏò2†- _wandbruntime‚6096  _runtime‚6096Þm‡çO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚217117811903488 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103005704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚7699 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚206539151197184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_st€Å7Mì$art/active.small_pool.allocated‚ 155696725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚230051038209024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10860724220928 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101532203 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43737064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155676687 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10860724220928 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101532201 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227978518121472 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59268106 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11049071096320 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚217368756732232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 257208888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚230051062149120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155676687 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚217117811903488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43737598 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10829423374284 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101532203 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚241099900817920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚217368574571468 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚206539168236544 _runtime‚6096.830919632 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚217117793900544 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155696725 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11048862608896 % _timestamp‚1.7884014721938531e+09»u*‚ò'ˆìO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11049071096320 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚217368756732232 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 257208888 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚230051062149120 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 155676687 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚217117811903488 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚43737598 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚10829423374284 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 101532203 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚241099900817920 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚217368574571468 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚206539168236544 _runtime‚6096.830919632 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚217117793900544 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 155696725 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11048862608896 % _timestamp‚1.7884014721938531e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚217117811903488 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103005704 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚7699 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚206539151197184 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 155696725 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚230051038209024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10860724220928 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 101532203 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚43737064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 155676687 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10860724220928 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 101532201 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚534 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚227978518121472 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚536 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚59268106 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408÷Y)"[2‰- _wandbruntime‚6097  _runtime‚6097rÏi«÷&ŠñM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚230079975097344 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚6097.505485904 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101544860 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103018368 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103017823 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43742885 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _step‚632 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10862117359104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 257241334 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 257241334 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59275481 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101544862 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155716538 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155696474 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚228007367640576 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚217395906918348 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚7699 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚217145063265280 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 257261400 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155696474 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101544860 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10862117359104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚217145081268224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚241130233291776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚217145081268224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43742342 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚241130467803136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257261400 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚230079951157248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚217145063265280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚206565111418880 % _timestamp‚1.7884014728681903e+09 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101544862ÅiD´Ö ‹öM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 155696474 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 101544860 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10862117359104 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚217145081268224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚241130233291776 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚217145081268224 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚43742342 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚241130467803136 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 257261400 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚230079951157248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚217145063265280 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚206565111418880 % _timestamp‚1.7884014728681903e+09 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 101544862 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚230079975097344 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚6097.505485904 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 101544860 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103018368 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103017823 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚43742885 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _step‚632 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10862117359104 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 257241334 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 257241334 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59275481 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 101544862 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 155716538 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 155696474 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚228007367640576 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚217395906918348 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚7699 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚217145063265280 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 257261400 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000øÅ°‰„2Œ- _wandbruntime‚6097  _runtime‚6097Ù!J\Ö epoch‚6 trainer/global_step‚7699 _runtime‚6097.506650061 /loss/FRt2/detection_loss‚2.9673377914088115 .loss/FRt4/detection_loss‚1.819108650307445 1loss/SRqqt1/detection_loss‚1.9661126329253118 0loss/FBt2/assignment_loss‚0.3737902735463447 /loss/FBt2/detection_loss‚2.8600242336591086 2loss/SRqqt3/assignment_loss‚1.5047069099151864 2loss/SRqqt4/assignment_loss‚0.9080642771835511 1loss/FBt3/assignment_loss‚0.20521760887155693 /loss/FRt1/detection_loss‚1.9352656222347702 .loss/FBt1/detection_loss‚2.669062701138583 %loss/total_loss‚43.71789483620705 .loss/FRt2/assignment_loss‚3.73069144146783 0loss/FRt3/assignment_loss‚3.1386757584718556 0loss/FBt1/assignment_loss‚0.3742718034562503 1loss/FBt4/assignment_loss‚0.19007638218896308 0loss/FRt4/assignment_loss‚1.8666077712002922 1loss/SRqqt2/assignment_loss‚1.817628325894475 1loss/SRqqt2/detection_loss‚2.5323240365833044 0loss/FBt4/detection_loss‚0.49278803118555753 _step‚633 /loss/FRt3/detection_loss‚2.4934691473459587 /loss/FBt3/detection_loss‚2.5911689549684525 1loss/SRqqt4/detection_loss‚1.1411186796541397 % _timestamp‚1.7884014728698773e+09 0loss/FRt1/assignment_loss‚2.8151644223502705 2loss/SRqqt1/assignment_loss‚1.4919320469877373 0loss/SRqqt3/detection_loss‚1.833287333261292hFaŽÛ 2loss/SRqqt3/assignment_loss‚1.5047069099151864 2loss/SRqqt4/assignment_loss‚0.9080642771835511 1loss/FBt3/assignment_loss‚0.20521760887155693 /loss/FRt1/detection_loss‚1.9352656222347702 .loss/FBt1/detection_loss‚2.669062701138583 %loss/total_loss‚43.71789483620705 .loss/FRt2/assignment_loss‚3.73069144146783 0loss/FRt3/assignment_loss‚3.1386757584718556 0loss/FBt1/assignment_loss‚0.3742718034562503 1loss/FBt4/assignment_loss‚0.19007638218896308 0loss/FRt4/assignment_loss‚1.8666077712002922 1loss/SRqqt2/assignment_loss‚1.817628325894475 1loss/SRqqt2/detection_loss‚2.5323240365833044 0loss/FBt4/detection_loss‚0.49278803118555753 _step‚633 /loss/FRt3/detection_loss‚2.4934691473459587 /loss/FBt3/detection_loss‚2.5911689549684525 1loss/SRqqt4/detection_loss‚1.1411186796541397 % _timestamp‚1.7884014728698773e+09 0loss/FRt1/assignment_loss‚2.8151644223502705 2loss/SRqqt1/assignment_loss‚1.4919320469877373 0loss/SRqqt3/detection_loss‚1.833287333261292 epoch‚6 trainer/global_step‚7699 _runtime‚6097.506650061 /loss/FRt2/detection_loss‚2.9673377914088115 .loss/FRt4/detection_loss‚1.819108650307445 1loss/SRqqt1/detection_loss‚1.9661126329253118 0loss/FBt2/assignment_loss‚0.3737902735463447 /loss/FBt2/detection_loss‚2.8600242336591086ùžÄôß’‚(:‡ È®ãÔªø–® gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚143.173' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.6346 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÙŸ‚‚(:÷ È®ãÔõ¹å° disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2603.15234375 cpu‚25.36366249311853 disk.rbd1.in‚0$ memory_percent‚94.74494457244873* proc.memory.percent‚15.888381004333496 network.sent‚ 1.8776104e+07 disk.drbd2080.in‚0 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd0.in‚0 network.recv‚ 2.172379e+06% disk./.usageGB‚229.51667404174805 disk.sda3.out‚ 7661.5390625 disk.rbd1.out‚ 14.9912109375 disk.drbd1716.in‚0 disk.dm-1.out‚ 7661.5390625* disk./.usagePercent‚15.403056572258189 disk.sda3.in‚ 0.05859375% disk.nvme0n1.out‚28543.8095703125 disk.sda2.in‚0 disk.dm-1.in‚ 0.05859375 disk.drbd2266.out‚0 disk.dm-0.in‚ 378.94921875 disk.nvme0n1.in‚ 378.94921875 disk.sda.in‚ 0.05859375 disk.drbd2266.in‚0 disk.drbd2265.out‚0" disk.dm-0.out‚28543.8095703125 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.sda.out‚ 7661.5390625' disk.nvme0c0n1.out‚28543.8095703125( proc.memory.availableMB‚ 860.98828125" disk.nvme0c0n1.in‚ 378.94921875zñYö’‘‚(:‡ ×®ãÔÛ¤¾® gpu.0.gpu‚24.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚151.023' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.2046 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚04ŸŠ?‚’‚(:÷ ×®ãÔ㸛±" disk.nvme0c0n1.in‚ 378.94921875 disk.rbd1.in‚0 disk.sda.out‚ 7664.88671875$ memory_percent‚94.74499225616455( proc.memory.availableMB‚ 860.98046875* disk./.usagePercent‚15.403059644348598% disk./.usageGB‚229.51671981811523! disk.dm-0.out‚28548.794921875 disk.sda1.in‚0 disk.nvme0n1.in‚ 378.94921875 disk.drbd1716.out‚0 network.sent‚ 1.888524e+07 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda.in‚ 0.05859375 disk.dm-1.in‚ 0.05859375 disk.dm-1.out‚ 7664.88671875 network.recv‚ 2.179133e+06 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 7664.88671875 disk.rbd1.out‚ 14.9912109375$ disk.nvme0n1.out‚28548.794921875 disk.drbd1716.in‚0 cpu‚25.362852465587814 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.sda1.out‚0& disk.nvme0c0n1.out‚28548.794921875 disk.sda2.in‚0# proc.memory.rssMB‚ 2603.15234375* proc.memory.percent‚15.888381004333496 disk.dm-0.in‚ 378.94921875 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.sda2.out‚0 disk.sda3.in‚ 0.05859375» hÑ2“- _wandbruntime‚6132  _runtime‚6132œÓëÆ”À & lr-AdamW/pg2‚7.393973760679336e-05 trainer/global_step‚7749 $ _timestamp‚1.788401507942895e+09 _runtime‚6132.579758437 _step‚634 & lr-AdamW/pg1‚7.393973760679336e-050Úϣ˕Štrainer/global_step‚7749 $ _timestamp‚1.788401507942895e+09 _runtime‚6132.579758437 _step‚634 & lr-AdamW/pg1‚7.393973760679336e-05 & lr-AdamW/pg2‚7.393973760679336e-05ú'“Ã2–- _wandbruntime‚6132  _runtime‚6132ECØèí'—çO F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102165151 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚218481280140288 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10930395543552 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚229411657680896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44008292 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10930395543552 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102165151 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚242616637576704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚218481262137344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚231496685620224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚242616403052032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚7749 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚229411657680896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102165153 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11119928016384 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 156689427 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 258854580 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚207836327353344 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚218481280140288 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10899060314096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103645248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 156669389 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚231496709560320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 % _timestamp‚1.7884015079435844e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040’†5Ÿ#˜ìO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚229411844683776 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚10930564543488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44007761 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚207836310313984 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚10899060314096 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 103645248 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 156669389 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚231496709560320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 % _timestamp‚1.7884015079435844e+09 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102165151 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚218481280140288 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚10930395543552 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚229411657680896 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44008292 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚10930395543552 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102165151 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚242616637576704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚218481262137344 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚231496685620224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚242616403052032 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚7749 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚229411657680896 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 102165153 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11119928016384 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 156689427 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 258854580 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚207836327353344 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocõ8ó,sated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚218481280140288 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10931788681728 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚218508549505024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102177812 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10900284357236 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10931788681728 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156709240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚218508531502080 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚207862253496320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚218508531502080 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258887052 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚218762537853556 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156709240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10931957694976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44013652 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 trainer/global_step‚7749 _runtime‚6133.259954316 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 $ _timestamp‚1.788401508622716e+09 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚207862270535680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚229440507200000 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚231525622508544 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 156689176 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10931957694976 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 258866986 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚218508549505024 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103657985 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44013161 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103657492 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0³<§Hû&›õM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚59644331 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11121139300352 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 102177810 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚491 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚218762720014528 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 258866986 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚10931788681728 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚218508549505024 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102177812 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10900284357236 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚10931788681728 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 156709240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚218508531502080 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚207862253496320 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚218508531502080 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 258887052 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚218762537853556 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 156709240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚10931957694976 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44013652 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 trainer/global_step‚7749 _runtime‚6133.259954316 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 $ _timestamp‚1.788401508622716e+09 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚207862270535680 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚229440507200000 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚231525622508544 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 156689176 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚10931957694976 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 258866986 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚218508549505024 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 103657985 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44013161 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 103657492 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 PDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230844797240320 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11000235866112 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232942356971520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚209133469430784 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102798101 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 260460192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102798101 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚219844730374144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚220102018602648 _step‚639 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚230844984243200 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 260480232 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 104285379 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚244132837110784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚232942333031424 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚219844730374144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚220101836441884 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 157682129 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚219844748377088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244133077926912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11000066866176 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104284864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60005804 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788401543691348e+09 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 157682129 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚230844797240320 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11000235866112 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚232942356971520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚209133469430784 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 102798101 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 260460192 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 102798101 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚219844730374144 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚220102018602648 _step‚639 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚230844984243200 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 260480232 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 104285379 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚244132837110784 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚232942333031424 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚219844730374144 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚220101836441884 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 157682129 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚219844748377088 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚244133077926912 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11000066866176 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104284864 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60005804 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 $ _timestamp‚1.788401543691348e+09ÿó4ÊW2«- _wandbruntime‚6169  _runtime‚6169*•û'ö&¬ðM [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11192130752000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11001460004352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚219872017741824 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚230873459743232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44285032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11001460004352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚244163400671744 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚230873646759424 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11001629017600 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260512704 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104297697 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60013183 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157701942 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚219871999738880 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260512704 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚209159412613120 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚220129168788764 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11001629017600 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚516 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚518 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102810762 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚640 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚220129350949736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10969756175644 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104298215 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚244163166160384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44284516 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚232971245979648üÒŠ­õM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚640 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚220129350949736 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚10969756175644 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104298215 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚244163166160384 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44284516 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚232971245979648 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11192130752000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11001460004352 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚219872017741824 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚230873459743232 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44285032 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11001460004352 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚244163400671744 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚230873646759424 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11001629017600 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 260512704 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104297697 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60013183 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 157701942 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚219871999738880 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 260512704 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚209159412613120 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚220129168788764 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11001629017600 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚516 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚518 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 102810762 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0€o­aˆ2®- _wandbruntime‚6169  _runtime‚6169Q3i\¯Ö 2loss/SRqqt3/assignment_loss‚1.7446366474032402 .loss/FRt3/detection_loss‚2.418777707768114 /loss/FBt4/detection_loss‚0.7173177928463379 trainer/global_step‚7799 2loss/SRqqt2/assignment_loss‚2.5682510137557983 1loss/FBt1/assignment_loss‚0.20333458051391062 /loss/FRt4/detection_loss‚1.9091306073522125 0loss/FRt4/assignment_loss‚1.6241731191122974 1loss/SRqqt4/detection_loss‚1.3632511518035944 1loss/FBt4/assignment_loss‚0.16552128142703965 1loss/SRqqt3/detection_loss‚1.5774397756904364 /loss/FBt1/detection_loss‚2.9645968794822695 .loss/FBt2/detection_loss‚2.231280451019605 2loss/SRqqt4/assignment_loss‚1.2625983823986293 0loss/SRqqt2/detection_loss‚2.358881611483438 epoch‚6 _runtime‚6169.002237712 /loss/FBt3/detection_loss‚2.6800868034362795 /loss/FRt2/detection_loss‚2.9867175136293684 0loss/FRt1/assignment_loss‚3.4830388736266356 /loss/FRt2/assignment_loss‚4.040210708975792 0loss/FBt2/assignment_loss‚0.5735200494527769 0loss/SRqqt1/detection_loss‚1.498527132366833 _step‚641 2loss/SRqqt1/assignment_loss‚1.7133983550756646 0loss/FBt3/assignment_loss‚0.2564772355835885 &loss/total_loss‚45.523958926360606 % _timestamp‚1.7884015443654563e+09 .loss/FRt1/detection_loss‚2.223960098738854 0loss/FRt3/assignment_loss‚2.9588311534178886 7þ´a°Û trainer/global_step‚7799 2loss/SRqqt2/assignment_loss‚2.5682510137557983 1loss/FBt1/assignment_loss‚0.20333458051391062 /loss/FRt4/detection_loss‚1.9091306073522125 0loss/FRt4/assignment_loss‚1.6241731191122974 1loss/SRqqt4/detection_loss‚1.3632511518035944 1loss/FBt4/assignment_loss‚0.16552128142703965 1loss/SRqqt3/detection_loss‚1.5774397756904364 /loss/FBt1/detection_loss‚2.9645968794822695 .loss/FBt2/detection_loss‚2.231280451019605 2loss/SRqqt4/assignment_loss‚1.2625983823986293 0loss/SRqqt2/detection_loss‚2.358881611483438 epoch‚6 _runtime‚6169.002237712 /loss/FBt3/detection_loss‚2.6800868034362795 /loss/FRt2/detection_loss‚2.9867175136293684 0loss/FRt1/assignment_loss‚3.4830388736266356 /loss/FRt2/assignment_loss‚4.040210708975792 0loss/FBt2/assignment_loss‚0.5735200494527769 0loss/SRqqt1/detection_loss‚1.498527132366833 _step‚641 2loss/SRqqt1/assignment_loss‚1.7133983550756646 0loss/FBt3/assignment_loss‚0.2564772355835885 &loss/total_loss‚45.523958926360606 % _timestamp‚1.7884015443654563e+09 .loss/FRt1/detection_loss‚2.223960098738854 0loss/FRt3/assignment_loss‚2.9588311534178886 2loss/SRqqt3/assignment_loss‚1.7446366474032402 .loss/FRt3/detection_loss‚2.418777707768114 /loss/FBt4/detection_loss‚0.7173177928463379P ’±‚(:‡ “¯ãԼͷ® gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚120.799' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.1598 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0èɫІ²‚(:û “¯ãÔÌÆ‘± disk.nvme0n1.in‚ 378.94921875 disk.drbd2265.in‚0 disk.sda1.out‚0 network.recv‚ 2.19762e+06 disk.sda3.in‚ 0.05859375 disk.sda3.out‚ 7666.86328125 disk.rbd1.out‚ 14.9912109375" disk.nvme0c0n1.in‚ 378.94921875& disk.nvme0n1.out‚28553.51416015625 disk.rbd0.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.05859375 disk.drbd2080.in‚0 disk.rbd0.out‚0* proc.memory.percent‚15.888381004333496 proc.cpu.threads‚17% disk./.usageGB‚229.51665496826172 disk.drbd2265.out‚0 disk.drbd2080.out‚0# disk.dm-0.out‚28553.51416015625& proc.memory.availableMB‚ 859.796875 network.sent‚ 1.9011602e+07( disk.nvme0c0n1.out‚28553.51416015625 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.rbd1.in‚0 disk.sda2.in‚0 disk.drbd2266.out‚0# proc.memory.rssMB‚ 2603.15234375 cpu‚25.359827936025134 disk.sda.in‚ 0.05859375 disk.drbd1716.out‚0 disk.dm-0.in‚ 378.94921875$ memory_percent‚94.75221633911133* disk./.usagePercent‚15.403055292220516 disk.sda.out‚ 7666.86328125 disk.dm-1.out‚ 7666.86328125 disk.sda1.in‚0mƒe³‚(:’ ¢¯ãÔà® gpu.0.gpu‚29.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚143.469' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.693799999999996 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÏBôˆ´‚(:ý ¢¯ãÔ¤‰ô¯ cpu‚25.359051033878476" disk.nvme0c0n1.in‚ 378.94921875 disk.nvme0n1.in‚ 378.94921875 disk.drbd1716.out‚0$ memory_percent‚94.74940299987793& disk.nvme0n1.out‚28553.98095703125 disk.sda.out‚ 7667.53515625 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.05859375 disk.rbd1.out‚ 14.9912109375 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0# proc.memory.rssMB‚ 2603.19140625 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.05859375 disk.dm-1.in‚ 0.05859375' proc.memory.availableMB‚ 860.2578125 network.sent‚ 1.9119822e+07 network.recv‚ 2.202711e+06* disk./.usagePercent‚15.403051452107503% disk./.usageGB‚229.51659774780273 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.dm-0.in‚ 378.94921875( disk.nvme0c0n1.out‚28553.98095703125 disk.dm-1.out‚ 7667.53515625# disk.dm-0.out‚28553.98095703125 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 7667.53515625* proc.memory.percent‚15.888619422912598 proc.cpu.threads‚17 disk.sda1.out‚01 ¨2µ- _wandbruntime‚6204  _runtime‚6204h[qŶ¿ % lr-AdamW/pg1‚7.10056273378745e-05 % lr-AdamW/pg2‚7.10056273378745e-05 trainer/global_step‚7849 % _timestamp‚1.7884015797240512e+09 _runtime‚6204.360895829 _step‚642ìG|Ê·Ä _runtime‚6204.360895829 _step‚642 % lr-AdamW/pg1‚7.10056273378745e-05 % lr-AdamW/pg2‚7.10056273378745e-05 trainer/global_step‚7849 % _timestamp‚1.7884015797240512e+09‚³TUÖ2¸- _wandbruntime‚6204  _runtime‚6204ÉÀÇ@í'¹çO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚519 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚232277936799744 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158674831 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11069907188736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44549848 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚210430628547584 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚232278123802624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚221208198610944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11261545837568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚221208198610944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262105884 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 _runtime‚6204.361874536 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158654793 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚245649309404160 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103431053 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 trainer/global_step‚7849 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103431053 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60374654 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11038003950912 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 262085844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚232277936799744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚245649550220288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚221468467377092 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103431051 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚234388004382720 _step‚643 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104924502 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11069738188800 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚517 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚210430645586944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158654793 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚232278123802624 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11261328961536 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199³8@…ò'ºìO H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 103431053 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 trainer/global_step‚7849 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 103431053 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚60374654 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11038003950912 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 262085844 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚232277936799744 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚245649550220288 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚221468467377092 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 103431051 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚234388004382720 _step‚643 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 104924502 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11069738188800 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚517 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚210430645586944 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 158654793 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚232278123802624 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11261328961536 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚519 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚232277936799744 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 158674831 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11069907188736 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44549848 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚210430628547584 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚232278123802624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚221208198610944 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11261545837568 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚221208198610944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 262105884 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 _runtime‚6204.361874536 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 158654793 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚245649309404160 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262138356 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚245679639589376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103443712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104937764 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚221495981884944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11039227994052 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚232306786318848 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11071300340224 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 158694644 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44555251 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 trainer/global_step‚7849 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60382033 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158674580 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6205.041673697 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 262118290 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚221495799723972 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104937282 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚232306786318848 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚221235467975680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103443710 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11262956769792 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103443712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚245679874100736 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11262746198528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚210456588769280 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158694644 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚221235485978624 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11071131326976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚210456571729920 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 262138356 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158674580 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚480[J'#ü&½öM =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11039393115664 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11071131326976 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 262118290 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11071300340224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚210456588769280 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 158694644 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚221235485978624 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11071131326976 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚210456571729920 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 262138356 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 158674580 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚480 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚232306599302656 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 262138356 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚245679639589376 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 103443712 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 104937764 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚482 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚221495981884944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11039227994052 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚232306786318848 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11071300340224 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 158694644 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44555251 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 trainer/global_step‚7849 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60382033 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 158674580 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚6205.041673697 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 262118290 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚221495799723972 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 104937282 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚232306786318848 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚221235467975680 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 103443710 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11262956769792 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 103443712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚245679874100736 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11262746198528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚233711076359168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 159647495 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚235833627853824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11332408026624 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚211727804703744 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104064003 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚233711263362048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚233711263362048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 159667533 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚247166059820544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884016152263262e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11139409511424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44820923 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105564427 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104064003 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11107475769320 _step‚647 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚222571666847744 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚222835280473064 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 263711496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚222835098312300 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚211727787664384‘KѲò'ÊìO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 159647495 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11139578511360 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚505 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚222571684850688 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚235833651793920 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104064001 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚222571666847744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚44821426 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _runtime‚6239.863597804 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 104064001 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚233711076359168 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 263731536 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚222571684850688 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11332197442048 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 159667533 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚247165825295872 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚233711263362048 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 159667533 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚247166059820544 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884016152263262e+09 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11139409511424 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚44820923 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 105564427 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 104064003 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11107475769320 _step‚647 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚222571666847744 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚222835280473064 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 263711496 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚222835098312300 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚211727787664384 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚233711076359168 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 159647495 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚235833627853824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11332408026624 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚211727804703744 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104064003 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚233711263362048 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚222862612820152 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚233739925878272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159667282 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44826685 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚222598954215424 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105577568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44826174 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚211753747886080 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚235862564742144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104076662 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104076662 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 159687346 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263743942 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228219904 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚235862540802048 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104076660 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚247196392360960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚222598954215424 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263764008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚211753730846720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11140971662848 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105577055 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159667282 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263764008 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚513 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11140802649600 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263743942 % _timestamp‚1.7884016158990684e+09 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 _runtime‚6240.536494143 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944{Žw8!ÏöM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚222598954215424 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 105577568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚44826174 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚211753747886080 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚235862564742144 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104076662 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 104076662 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 159687346 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 263743942 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228219904 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚235862540802048 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104076660 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚247196392360960 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚222598954215424 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 263764008 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚211753730846720 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11140971662848 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 105577055 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 159667282 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 263764008 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚513 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11140802649600 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 263743942 % _timestamp‚1.7884016158990684e+09 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.currenÛïUÄt‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 _runtime‚6240.536494143 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚60750883 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11140802649600 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚222598936212480 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚222862430659180 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚222862612820152 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚233739925878272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 159667282 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚44826685ˆÞÅ­2Ð- _wandbruntime‚6240  _runtime‚6240ÉÀ„ ]Ñ× 0loss/FRt4/assignment_loss‚1.2588608860969543 2loss/SRqqt4/assignment_loss‚1.0221925348540146 2loss/SRqqt2/assignment_loss‚2.2260315964619317 .loss/FRt4/detection_loss‚1.819987729825872 /loss/FRt2/assignment_loss‚4.315110240131617 2loss/SRqqt3/assignment_loss‚1.5941752587380105 1loss/FBt3/assignment_loss‚0.16334322135653423 .loss/FRt2/detection_loss‚3.072821009904146 /loss/FRt1/assignment_loss‚3.169765972665378 /loss/FBt1/detection_loss‚2.0778517169611797 epoch‚6 trainer/global_step‚7899 % _timestamp‚1.7884016159007897e+09 1loss/FBt1/assignment_loss‚0.43215325218605877 /loss/FRt3/detection_loss‚2.6904518801358437 1loss/SRqqt3/detection_loss‚2.0037127797340237 /loss/FBt2/detection_loss‚2.7119200030962625 .loss/FBt3/detection_loss‚2.871078312397003 _step‚649 0loss/FRt3/assignment_loss‚3.3905384848194737 2loss/SRqqt1/assignment_loss‚1.5212453410290836 1loss/FBt4/assignment_loss‚0.13823984417597066 1loss/SRqqt2/detection_loss‚2.4676124731699627 1loss/SRqqt4/detection_loss‚0.9903298469384512 /loss/FBt4/detection_loss‚0.7228164902221641 /loss/FRt1/detection_loss‚1.9510189103229127 1loss/SRqqt1/detection_loss‚1.6525831742767703 %loss/total_loss‚44.78922684437815 _runtime‚6240.537544725 0loss/FBt2/assignment_loss‚0.5253858848785361‚Ù bÒÜ 2loss/SRqqt2/assignment_loss‚2.2260315964619317 .loss/FRt4/detection_loss‚1.819987729825872 /loss/FRt2/assignment_loss‚4.315110240131617 2loss/SRqqt3/assignment_loss‚1.5941752587380105 1loss/FBt3/assignment_loss‚0.16334322135653423 .loss/FRt2/detection_loss‚3.072821009904146 /loss/FRt1/assignment_loss‚3.169765972665378 /loss/FBt1/detection_loss‚2.0778517169611797 epoch‚6 trainer/global_step‚7899 % _timestamp‚1.7884016159007897e+09 1loss/FBt1/assignment_loss‚0.43215325218605877 /loss/FRt3/detection_loss‚2.6904518801358437 1loss/SRqqt3/detection_loss‚2.0037127797340237 /loss/FBt2/detection_loss‚2.7119200030962625 .loss/FBt3/detection_loss‚2.871078312397003 _step‚649 0loss/FRt3/assignment_loss‚3.3905384848194737 2loss/SRqqt1/assignment_loss‚1.5212453410290836 1loss/FBt4/assignment_loss‚0.13823984417597066 1loss/SRqqt2/detection_loss‚2.4676124731699627 1loss/SRqqt4/detection_loss‚0.9903298469384512 /loss/FBt4/detection_loss‚0.7228164902221641 /loss/FRt1/detection_loss‚1.9510189103229127 1loss/SRqqt1/detection_loss‚1.6525831742767703 %loss/total_loss‚44.78922684437815 _runtime‚6240.537544725 0loss/FBt2/assignment_loss‚0.5253858848785361 0loss/FRt4/assignment_loss‚1.2588608860969543 2loss/SRqqt4/assignment_loss‚1.0221925348540146‰™g¾DÓ‚(:’ Þ¯ãÔ·€ð­ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚133.197' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.639400000000002 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îI7ò†Ô‚(:û Þ¯ãÔ¬ø‚± disk.drbd2266.in‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚28555.41552734375 disk.nvme0n1.in‚ 378.94921875 disk.drbd1716.out‚0# proc.memory.rssMB‚ 2603.19140625 proc.cpu.threads‚17) disk./.usagePercent‚15.40305478020545 disk.drbd2080.out‚0# disk.dm-0.out‚28555.41552734375 disk.sda3.in‚ 0.05859375 disk.sda2.out‚0* proc.memory.percent‚15.888619422912598 network.sent‚ 1.9337571e+07 disk.dm-1.out‚ 7669.41015625 disk.drbd2265.out‚0 disk.sda3.out‚ 7669.41015625$ disk./.usageGB‚229.5166473388672 disk.sda.out‚ 7669.41015625# memory_percent‚94.7500467300415 disk.dm-1.in‚ 0.05859375 disk.dm-0.in‚ 378.94921875 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 378.94921875 disk.drbd2265.in‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 network.recv‚ 2.223293e+06 disk.rbd0.out‚0 disk.rbd1.out‚ 14.9912109375& disk.nvme0n1.out‚28555.41552734375( proc.memory.availableMB‚ 860.15234375 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.05859375 cpu‚25.356088992939867þ¦q’Õ‚(:‡ í¯ãÔŽí‚® gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚117.022' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4044 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lèþÖ‚(:ö í¯ãÔ؛ܱ( disk.nvme0c0n1.out‚28567.35791015625 disk.drbd1716.out‚0 disk.rbd0.out‚0 disk.sda2.out‚0 disk.drbd1716.in‚0# proc.memory.rssMB‚ 2603.19140625# disk.dm-0.out‚28567.35791015625* proc.memory.percent‚15.888619422912598 network.recv‚ 2.229523e+06* disk./.usagePercent‚15.403496649209547 disk.sda3.out‚ 7673.1796875 disk.nvme0n1.in‚ 378.94921875& disk.nvme0n1.out‚28567.35791015625 disk.sda.in‚ 0.05859375 disk.dm-1.out‚ 7673.1796875 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 378.94921875 disk.sda3.in‚ 0.05859375 disk.rbd1.in‚0 disk.drbd2265.in‚0 network.sent‚ 1.943567e+07 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.05859375 disk.drbd2265.out‚0$ memory_percent‚94.75154876708984 cpu‚25.35535383282425% disk./.usageGB‚229.52323150634766 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 14.9912109375 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 378.94921875 disk.sda.out‚ 7673.1796875 disk.drbd2266.out‚0% proc.memory.availableMB‚ 859.90625•&|¼2×- _wandbruntime‚6275  _runtime‚6275 Ú2qÆØÀ & lr-AdamW/pg1‚6.810193139515827e-05 & lr-AdamW/pg2‚6.810193139515827e-05 trainer/global_step‚7949 % _timestamp‚1.7884016510249567e+09 _runtime‚ 6275.66182032 _step‚650â4d™ËÙÅ & lr-AdamW/pg2‚6.810193139515827e-05 trainer/global_step‚7949 % _timestamp‚1.7884016510249567e+09 _runtime‚ 6275.66182032 _step‚650 & lr-AdamW/pg1‚6.810193139515827e-05ŠÒ0Â2Ú- _wandbruntime‚6275  _runtime‚6275ý;Ögí'ÛçO H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104696951 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160660235 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚223935153087488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11176782466324 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚235144402921472 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265357188 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11209249833984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚223935153087488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚532 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160640197 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚223935135084544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚224201729247508 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104696953 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106205227 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11403039639040 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚223935135084544 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚7949 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61112354 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚237279299205120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚651 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11209080834048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235144215918592 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160660235 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11403256515072 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚213024946781184 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚534 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11209080834048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚235144402921472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45092873 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚248682314904064 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11176947587728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚224201911408272 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235144215918592 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61112356 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45093405 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚237279275265024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 % _timestamp‚1.7884016510257714e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0°Éå”ò'ÜìO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚235144215918592 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61112356 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45093405 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚237279275265024 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 % _timestamp‚1.7884016510257714e+09 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 104696951 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 160660235 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚223935153087488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11176782466324 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚235144402921472 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 265357188 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11209249833984 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚223935153087488 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚532 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 160640197 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚223935135084544 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚224201729247508 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 104696953 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106205227 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11403039639040 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚223935135084544 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 trainer/global_step‚7949 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61112354 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚237279299205120 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _step‚651 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11209080834048 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚235144215918592 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 160660235 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11403256515072 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚213024946781184 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚534 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11209080834048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚235144402921472 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45092873 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚248682314904064 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11176947587728 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚224201911408272 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257‹œúî2Ý- _wandbruntime‚6276  _runtime‚6276•„æŸÞñM D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚224229061594388 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104709610 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106218394 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚237308188213248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11178336752480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11178171630868 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104709612 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104709610 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚213050907002880 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 _step‚652 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚248712647820288 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚235173065437696 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 % _timestamp‚1.7884016516968455e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45098663 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 160680048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚223962422452224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160659984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚223962404449280 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚235172878421504 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106218920 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265389660 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚235173065437696 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265389660 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 265369594 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11404459607040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _runtime‚6276.334367158 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 265369594 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096ŠôÃ5ü&ßöM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 265369594 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚224229061594388 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 104709610 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106218394 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚237308188213248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11178336752480 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11178171630868 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 104709612 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 104709610 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚213050907002880 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 _step‚652 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚248712647820288 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚235173065437696 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 % _timestamp‚1.7884016516968455e+09 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45098663 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 160680048 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚223962422452224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 160659984 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚223962404449280 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚235172878421504 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106218920 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 265389660 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚235173065437696 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 265389660 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 265369594 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11404459607040 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _runtime‚6276.334367158 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217ŒØ™â2à- _wandbruntime‚6276  _runtime‚6276#«H†VáÐ /loss/FRt1/detection_loss‚2.0792119646513934 .loss/FRt3/assignment_loss‚2.68119136265346 2loss/SRqqt3/assignment_loss‚1.4734273400778573 0loss/FBt2/assignment_loss‚0.3461857249896688 .loss/FRt3/detection_loss‚2.390080577135086 /loss/FRt4/detection_loss‚2.1746994622729043 /loss/FRt4/assignment_loss‚1.775795186107809 2loss/SRqqt2/assignment_loss‚2.1520633413814583 1loss/FBt4/assignment_loss‚0.11777542912319172 epoch‚6 /loss/FRt1/assignment_loss‚3.308539741569095 2loss/SRqqt1/assignment_loss‚1.3550905521426881 0loss/FBt3/assignment_loss‚0.2440572369377918 .loss/FRt2/detection_loss‚3.002338773929156 trainer/global_step‚7949 0loss/FBt1/assignment_loss‚0.1231832472451894 1loss/SRqqt2/detection_loss‚2.7066363256711226 .loss/FBt3/detection_loss‚4.539369642734528 %loss/total_loss‚47.06218320016596 2loss/SRqqt4/assignment_loss‚1.3393437773721026 /loss/SRqqt1/detection_loss‚1.96341515195613 1loss/SRqqt4/detection_loss‚0.9831289116825376 .loss/FBt2/detection_loss‚2.554220634348252 /loss/FBt4/detection_loss‚0.7619560505928737 _runtime‚6276.335331918 _step‚653 0loss/FRt2/assignment_loss‚3.3403179874786963 1loss/SRqqt3/detection_loss‚1.8867618925869465 /loss/FBt1/detection_loss‚3.7633928855260215 % _timestamp‚1.7884016516985326e+09Sþ[âÕ 2loss/SRqqt4/assignment_loss‚1.3393437773721026 /loss/SRqqt1/detection_loss‚1.96341515195613 1loss/SRqqt4/detection_loss‚0.9831289116825376 .loss/FBt2/detection_loss‚2.554220634348252 /loss/FBt4/detection_loss‚0.7619560505928737 _runtime‚6276.335331918 _step‚653 0loss/FRt2/assignment_loss‚3.3403179874786963 1loss/SRqqt3/detection_loss‚1.8867618925869465 /loss/FBt1/detection_loss‚3.7633928855260215 % _timestamp‚1.7884016516985326e+09 /loss/FRt1/detection_loss‚2.0792119646513934 .loss/FRt3/assignment_loss‚2.68119136265346 2loss/SRqqt3/assignment_loss‚1.4734273400778573 0loss/FBt2/assignment_loss‚0.3461857249896688 .loss/FRt3/detection_loss‚2.390080577135086 /loss/FRt4/detection_loss‚2.1746994622729043 /loss/FRt4/assignment_loss‚1.775795186107809 2loss/SRqqt2/assignment_loss‚2.1520633413814583 1loss/FBt4/assignment_loss‚0.11777542912319172 epoch‚6 /loss/FRt1/assignment_loss‚3.308539741569095 2loss/SRqqt1/assignment_loss‚1.3550905521426881 0loss/FBt3/assignment_loss‚0.2440572369377918 .loss/FRt2/detection_loss‚3.002338773929156 trainer/global_step‚7949 0loss/FBt1/assignment_loss‚0.1231832472451894 1loss/SRqqt2/detection_loss‚2.7066363256711226 .loss/FBt3/detection_loss‚4.539369642734528 %loss/total_loss‚47.06218320016596éÿ&õ’ã‚(:‡ ü¯ãÔø·®® gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚141.577' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.3154 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0\œkä‚(:à ü¯ãÔùγ± network.recv‚ 2.233638e+06$ disk./.usageGB‚229.5231819152832 disk.drbd1716.in‚0 cpu‚25.35459090921149 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 network.sent‚ 1.944159e+07 disk.nvme0n1.in‚ 378.94921875 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.05859375 disk.drbd2265.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2603.19140625* proc.memory.percent‚15.888619422912598" disk.nvme0n1.out‚ 28573.1015625 disk.sda3.out‚7676.375 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 378.94921875 disk.dm-1.out‚7676.375$ memory_percent‚94.75023746490479( proc.memory.availableMB‚ 860.12109375 disk.rbd1.out‚ 14.9912109375 disk.dm-0.in‚ 378.94921875 proc.cpu.threads‚17* disk./.usagePercent‚15.403493321111602 disk.sda.in‚ 0.05859375 disk.sda.out‚7676.375 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.dm-0.out‚ 28573.1015625 disk.sda1.out‚0$ disk.nvme0c0n1.out‚ 28573.1015625 disk.dm-1.in‚ 0.05859375 disk.rbd0.in‚0¾´DLå‚(:’ ‹°ãÔÒ×ù­ gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚143.998' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.799599999999998 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚07诅æ‚(:ú ‹°ãÔÿóâ± disk.dm-1.out‚ 7677.8984375 disk.dm-0.in‚ 378.94921875 disk.sda1.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0( proc.memory.availableMB‚ 860.33984375* proc.memory.percent‚15.888619422912598 disk.drbd2266.out‚0$ disk./.usageGB‚229.5222511291504 disk.sda3.out‚ 7677.8984375 disk.rbd1.in‚0( disk.nvme0c0n1.out‚28577.13330078125 disk.drbd1716.in‚0 disk.drbd1716.out‚0 network.sent‚ 1.9549818e+07 disk.nvme0n1.in‚ 378.94921875 network.recv‚ 2.239753e+06 disk.sda3.in‚ 0.05859375 cpu‚25.353865382935425 proc.cpu.threads‚17 disk.sda1.out‚0 disk.rbd1.out‚ 14.9912109375& disk.nvme0n1.out‚28577.13330078125 disk.sda.in‚ 0.05859375 disk.sda2.out‚0$ memory_percent‚94.74890232086182# proc.memory.rssMB‚ 2603.19140625 disk.drbd2080.in‚0" disk.nvme0c0n1.in‚ 378.94921875 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.sda.out‚ 7677.8984375 disk.sda2.in‚0 disk.dm-1.in‚ 0.05859375 disk.drbd2080.out‚0# disk.dm-0.out‚28577.13330078125* disk./.usagePercent‚15.403430855273248¤ÇÀ12ç- _wandbruntime‚6311  _runtime‚6311Ûy‰ÝÅè¿ & lr-AdamW/pg1‚6.666224620969634e-05 & lr-AdamW/pg2‚6.666224620969634e-05 trainer/global_step‚7999 $ _timestamp‚1.788401686819339e+09 _runtime‚ 6311.45627236 _step‚654¼ü„ÊÊéÄ _step‚654 & lr-AdamW/pg1‚6.666224620969634e-05 & lr-AdamW/pg2‚6.666224620969634e-05 trainer/global_step‚7999 $ _timestamp‚1.788401686819339e+09 _runtime‚ 6311.45627236Ž&3ŒO2ê- _wandbruntime‚6311  _runtime‚6311J¬Îì'ëæO E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11278752156672 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 161652937 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚7999 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚236577542480896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106846323 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚238724922676224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚214322105897984 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚6311.457091636 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚236577355478016 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105329901 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 161632899 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 $ _timestamp‚1.788401686820056e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚250198866292224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _step‚655 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11246419406136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 161632899 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11278921156608 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 266962800 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚225298603321344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 266982840 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105329903 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61481204 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚214322122937344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚238724946616320 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11473943616000 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 46242201609‰ŒîbìëO WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚225298603321344 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360#Ú& UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚236577355478016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚225568360182716 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11474154200576 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 106846882 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚557 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45365676 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45365119 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 161632899 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11278921156608 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 266962800 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚225298603321344 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 266982840 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105329903 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚61481204 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚214322122937344 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚238724946616320 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11473943616000 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11278752156672 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 161652937 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 trainer/global_step‚7999 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚236577542480896 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 106846323 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚238724922676224 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚214322105897984 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 _runtime‚6311.457091636 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚236577355478016 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105329901 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 161632899 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 $ _timestamp‚1.788401686820056e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚250198866292224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _step‚655 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11246419406136 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11280145294848 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11280145294848 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚656 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚236606204997120 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105342562 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45371152 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚250229432687616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105342560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚214348066119680 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11475573123072 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚225325872686080 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106859218 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11247808570888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚515 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106859735 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚517 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105342560 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚236606017980928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚214348049080320 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 267015312 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚225325872686080 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 266995246 _runtime‚6312.136902106 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚236606204997120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267015312 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11247643449276 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 161652686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45370637 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚225595874690568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0_Œ6,û&ïõM bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11475573123072 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚225325872686080 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 106859218 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11247808570888 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚515 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 106859735 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚517 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105342560 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚236606017980928 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚214348049080320 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 267015312 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚225325872686080 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 266995246 _runtime‚6312.136902106 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚236606204997120 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 267015312 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11247643449276 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 161652686 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45370637 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚225595874690568 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61488583 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚238753859564544 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11280145294848 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11280145294848 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚656 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚236606204997120 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 105342562 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45371152 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚250229432687616 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105342560 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚214348066119680 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416RƵ…2ð- _wandbruntime‚6312  _runtime‚6312cü0kVñÐ 0loss/FBt1/assignment_loss‚0.6526532651390881 /loss/FRt3/detection_loss‚2.5991757206014685 0loss/SRqqt2/detection_loss‚2.787334907622564 1loss/SRqqt4/detection_loss‚1.0647550137364306 % _timestamp‚1.7884016875015616e+09 1loss/SRqqt1/assignment_loss‚1.670445454996769 0loss/FBt3/assignment_loss‚0.3632982370327227 1loss/SRqqt1/detection_loss‚1.9454391731156244 epoch‚6 2loss/SRqqt4/assignment_loss‚0.7911937839235179 1loss/FBt4/assignment_loss‚0.18762810949621733 /loss/FRt4/detection_loss‚1.8758515065839125 /loss/FBt3/detection_loss‚2.7225564643740654 /loss/FRt2/assignment_loss‚3.593116589954921 1loss/SRqqt2/assignment_loss‚2.351620179556665 .loss/FBt1/detection_loss‚2.246233916282654 _step‚657 .loss/FRt1/detection_loss‚1.989356031784646 .loss/FRt2/detection_loss‚3.537460588273548 1loss/SRqqt3/detection_loss‚2.2201460237438613 trainer/global_step‚7999 _runtime‚6312.138326394 0loss/FRt1/assignment_loss‚3.5657877727196765 .loss/FBt2/detection_loss‚4.549376058578491 %loss/total_loss‚48.26336439187882 /loss/FRt3/assignment_loss‚3.317908306379576 0loss/FBt2/assignment_loss‚0.3629066992121793 /loss/FBt4/detection_loss‚0.6663934357486487 0loss/FRt4/assignment_loss‚1.9652056735615397 1loss/SRqqt3/assignment_loss‚1.237521479460033Ûfb`[òÕ 1loss/FBt4/assignment_loss‚0.18762810949621733 /loss/FRt4/detection_loss‚1.8758515065839125 /loss/FBt3/detection_loss‚2.7225564643740654 /loss/FRt2/assignment_loss‚3.593116589954921 1loss/SRqqt2/assignment_loss‚2.351620179556665 .loss/FBt1/detection_loss‚2.246233916282654 _step‚657 .loss/FRt1/detection_loss‚1.989356031784646 .loss/FRt2/detection_loss‚3.537460588273548 1loss/SRqqt3/detection_loss‚2.2201460237438613 trainer/global_step‚7999 _runtime‚6312.138326394 0loss/FRt1/assignment_loss‚3.5657877727196765 .loss/FBt2/detection_loss‚4.549376058578491 %loss/total_loss‚48.26336439187882 /loss/FRt3/assignment_loss‚3.317908306379576 0loss/FBt2/assignment_loss‚0.3629066992121793 /loss/FBt4/detection_loss‚0.6663934357486487 0loss/FRt4/assignment_loss‚1.9652056735615397 1loss/SRqqt3/assignment_loss‚1.237521479460033 0loss/FBt1/assignment_loss‚0.6526532651390881 /loss/FRt3/detection_loss‚2.5991757206014685 0loss/SRqqt2/detection_loss‚2.787334907622564 1loss/SRqqt4/detection_loss‚1.0647550137364306 % _timestamp‚1.7884016875015616e+09 1loss/SRqqt1/assignment_loss‚1.670445454996769 0loss/FBt3/assignment_loss‚0.3632982370327227 1loss/SRqqt1/detection_loss‚1.9454391731156244 epoch‚6 2loss/SRqqt4/assignment_loss‚0.7911937839235179‘ÉØnc3ó‚(:‡ š°ãÔ–¥¢¯ gpu.0.gpu‚31.0 gpÚ¾v¿_u.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚131.841' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.3682 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0lne–ô‚(:ô š°ãÔžÞ€° disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0% disk.nvme0c0n1.out‚28607.05078125# proc.memory.rssMB‚ 2603.19140625* proc.memory.percent‚15.888619422912598 disk.sda2.in‚0 disk.dm-1.in‚ 0.05859375 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.05859375 disk.sda.in‚ 0.05859375 disk.sda.out‚ 7682.13671875$ memory_percent‚94.74883079528809 network.sent‚ 1.9555153e+07 network.recv‚ 2.243343e+06 disk.sda3.out‚ 7682.13671875% disk./.usageGB‚229.52816772460938 disk.nvme0n1.in‚ 378.98046875 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0 cpu‚25.353121749598568 proc.cpu.threads‚17# disk.nvme0n1.out‚28607.05078125 disk.drbd1716.out‚0 disk.dm-1.out‚ 7682.13671875 disk.drbd2265.out‚0 disk.dm-0.out‚28607.05078125" disk.nvme0c0n1.in‚ 378.98046875* disk./.usagePercent‚15.403827922958854 disk.drbd2080.in‚0 disk.dm-0.in‚ 378.98046875 disk.sda1.in‚0 disk.rbd1.out‚ 14.9912109375' proc.memory.availableMB‚ 860.3515625¨“‘õ‚(:„ ©°ãÔÜáñ­ gpu.0.gpu‚30.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚146.95' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.39 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0—EÐ.zö‚(:ï ©°ãÔ¯ßDZ& proc.memory.availableMB‚ 860.046875 cpu‚25.35239334391473 disk.nvme0n1.in‚ 378.98046875# disk.nvme0n1.out‚28607.31640625 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 15.0771484375* disk./.usagePercent‚15.403826642921183 disk.dm-1.out‚ 7682.8984375 disk.sda3.out‚ 7682.8984375 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0% disk./.usageGB‚229.52814865112305 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 378.98046875# proc.memory.rssMB‚ 2603.19140625* proc.memory.percent‚15.888619422912598 disk.sda.out‚ 7682.8984375$ memory_percent‚94.75069046020508 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.05859375" disk.nvme0c0n1.in‚ 378.98046875 network.sent‚ 1.9663836e+07 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.05859375 disk.drbd2265.out‚0 disk.dm-0.out‚28607.31640625 network.recv‚ 2.249329e+06 disk.sda.in‚ 0.05859375 disk.sda2.in‚0% disk.nvme0c0n1.out‚28607.31640625·¢’÷‚(:‡ ¸°ãÔÊÛº® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚131.841' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.3682 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õ/ëú‚ø‚(:÷ ¸°ãÔÑï° network.sent‚ 1.9669835e+07 disk.sda1.in‚0 disk.rbd1.out‚ 15.1396484375" disk.nvme0c0n1.in‚ 378.98046875 disk.sda.out‚ 7683.703125 disk.drbd1716.out‚0# proc.memory.rssMB‚ 2603.19140625 disk.drbd2265.in‚0 disk.drbd2265.out‚0* disk./.usagePercent‚15.403830995049264( disk.nvme0c0n1.out‚28609.06396484375' proc.memory.availableMB‚ 860.2734375 disk.dm-0.in‚ 378.98046875 disk.sda3.in‚ 0.05859375 disk.sda3.out‚ 7683.703125 disk.nvme0n1.in‚ 378.98046875 disk.dm-1.in‚ 0.05859375 proc.cpu.threads‚17# disk.dm-0.out‚28609.06396484375 disk.sda.in‚ 0.05859375 disk.drbd2080.out‚0* proc.memory.percent‚15.888619422912598 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 7683.703125 disk.rbd0.in‚0 cpu‚25.351647701424877 disk.drbd2266.in‚0 network.recv‚ 2.253656e+06% disk./.usageGB‚229.52821350097656 disk.sda1.out‚0& disk.nvme0n1.out‚28609.06396484375 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda2.in‚0 disk.drbd2266.out‚0$ memory_percent‚94.74930763244629¯Ó¾2ù- _wandbruntime‚6347  _runtime‚6347fm3ÇúÁ _runtime‚6347.258387581 _step‚658 & lr-AdamW/pg1‚6.523107015606654e-05 & lr-AdamW/pg2‚6.523107015606654e-05 trainer/global_step‚8049 % _timestamp‚1.7884017226215458e+09ßÿU€ÌûÆ & lr-AdamW/pg1‚6.523107015606654e-05 & lr-AdamW/pg2‚6.523107015606654e-05 trainer/global_step‚8049 % _timestamp‚1.7884017226215458e+09 _runtime‚6347.258387581 _step‚658’3Jß2ü- _wandbruntime‚6347  _runtime‚6347òÆ•í'ýçO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 268608492 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11544976757248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚238010495037440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105962853 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 162625601 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚226662071558144 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11315891224544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚238010682040320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚226662089561088 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162645639 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11348423479296 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11348592479232 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238010495037440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚226662071558144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105962851 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45637095 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11315726103140 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 trainer/global_step‚8049 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚251715336260096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11544766172672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚226934991117924 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _runtime‚6347.259331462 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 162645639 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚659 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107486616 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11348423479296 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚215619265014784 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 % _timestamp‚1.7884017226222491e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61850056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248ÓDöò'þìO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚226662089561088 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 162645639 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11348423479296 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11348592479232 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚238010495037440 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚226662071558144 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 105962851 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚45637095 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11315726103140 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 trainer/global_step‚8049 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚251715336260096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11544766172672 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚226934991117924 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _runtime‚6347.259331462 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 162645639 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _step‚659 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 107486616 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11348423479296 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚215619265014784 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 % _timestamp‚1.7884017226222491e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚61850056 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 268608492 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11544976757248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚238010495037440 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚535 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 105962853 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 162625601 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚226662071558144 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚533 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11315891224544 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚238010682040320 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚226689358925824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61857433 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162645388 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11317115267684 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚238039157540352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45641910 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚240199483035648 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚238039344556544 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107499854 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11546184885248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 268640964 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚226962323464804 _runtime‚6347.930416338 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61857431 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚215645208197120 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 268640964 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162665452 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚226689340922880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11349816617472 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚240199506975744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 trainer/global_step‚8049 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105975510 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107499341 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚251745900335104 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45642421 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚251745667920896 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚226962505625776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11349985630720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416Uì^ìü& öM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 107499854 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11546184885248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 268640964 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚226962323464804 _runtime‚6347.930416338 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚61857431 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚215645208197120 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 268640964 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 162665452 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚226689340922880 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11349816617472 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚240199506975744 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 trainer/global_step‚8049 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 105975510 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 107499341 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚251745900335104 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45642421 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚251745667920896 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚226962505625776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11349985630720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 105975510 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚226689340922880 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚226689358925824 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚61857433 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 162645388 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11317115267684 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚238039157540352 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45641910 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚240199483035648 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚238039344556544”Guó¦2‚ - _wandbruntime‚6347  _runtime‚6347Ó—†•[ƒ Õ /loss/FBt1/detection_loss‚2.5052176316579184 0loss/FRt1/assignment_loss‚2.8539409104146456 0loss/FRt4/assignment_loss‚1.7488035783171654 1loss/SRqqt1/assignment_loss‚1.583621060712771 1loss/SRqqt3/assignment_loss‚1.405713443653131 1loss/SRqqt1/detection_loss‚1.7358363184061918 1loss/SRqqt3/detection_loss‚1.8691540470100798 &loss/total_loss‚44.068950472635436 0loss/FBt3/assignment_loss‚0.1490412669788514 /loss/FRt1/detection_loss‚2.0942027009370032 _runtime‚6347.931529979 2loss/SRqqt4/assignment_loss‚0.8682898828952477 /loss/FRt4/detection_loss‚1.7508942214772105 0loss/SRqqt2/detection_loss‚2.259415328502655 0loss/FBt1/assignment_loss‚0.4297297691097002 1loss/FBt4/assignment_loss‚0.19185751421544311 /loss/FRt2/detection_loss‚2.8989043271902837 1loss/SRqqt4/detection_loss‚1.0457614797071135 trainer/global_step‚8049 % _timestamp‚1.7884017232947934e+09 /loss/FRt2/assignment_loss‚4.103094346595533 0loss/FRt3/assignment_loss‚3.0122901671811153 2loss/SRqqt2/assignment_loss‚2.1646223089524677 /loss/FRt3/detection_loss‚2.4869365645082375 .loss/FBt2/detection_loss‚2.936726620564094 epoch‚6 0loss/FBt2/assignment_loss‚0.2085447694001209 _step‚661 .loss/FBt3/detection_loss‚3.115810419831957 /loss/FBt4/detection_loss‚0.6505417944164946¸"±,`„ Ú % _timestamp‚1.7884017232947934e+09 /loss/FRt2/assignment_loss‚4.103094346595533 0loss/FRt3/assignment_loss‚3.0122901671811153 2loss/SRqqt2/assignment_loss‚2.1646223089524677 /loss/FRt3/detection_loss‚2.4869365645082375 .loss/FBt2/detection_loss‚2.936726620564094 epoch‚6 0loss/FBt2/assignment_loss‚0.2085447694001209 _step‚661 .loss/FBt3/detection_loss‚3.115810419831957 /loss/FBt4/detection_loss‚0.6505417944164946 /loss/FBt1/detection_loss‚2.5052176316579184 0loss/FRt1/assignment_loss‚2.8539409104146456 0loss/FRt4/assignment_loss‚1.7488035783171654 1loss/SRqqt1/assignment_loss‚1.583621060712771 1loss/SRqqt3/assignment_loss‚1.405713443653131 1loss/SRqqt1/detection_loss‚1.7358363184061918 1loss/SRqqt3/detection_loss‚1.8691540470100798 &loss/total_loss‚44.068950472635436 0loss/FBt3/assignment_loss‚0.1490412669788514 /loss/FRt1/detection_loss‚2.0942027009370032 _runtime‚6347.931529979 2loss/SRqqt4/assignment_loss‚0.8682898828952477 /loss/FRt4/detection_loss‚1.7508942214772105 0loss/SRqqt2/detection_loss‚2.259415328502655 0loss/FBt1/assignment_loss‚0.4297297691097002 1loss/FBt4/assignment_loss‚0.19185751421544311 /loss/FRt2/detection_loss‚2.8989043271902837 1loss/SRqqt4/detection_loss‚1.0457614797071135 trainer/global_step‚8049•õ'ôõ’… ‚(:‡ Ç°ãÔÛ® gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚120.113' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.0226 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ßû9ˆ† ‚(:ý Ç°ãÔ¼„¢² disk.drbd2080.in‚0 disk.sda3.out‚ 7684.17578125 disk.rbd1.in‚0# proc.memory.rssMB‚ 2603.19140625* disk./.usagePercent‚15.403834835162279 disk.sda.in‚ 0.05859375 disk.dm-1.out‚ 7684.17578125 disk.drbd2265.out‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚28609.24365234375* proc.memory.percent‚15.888619422912598 network.recv‚ 2.258294e+06" disk.nvme0c0n1.in‚ 378.98046875 disk.nvme0n1.in‚ 378.98046875 disk.sda2.out‚0 disk.dm-1.in‚ 0.05859375 disk.drbd2265.in‚0 disk.drbd2080.out‚0# disk.dm-0.out‚28609.24365234375% disk./.usageGB‚229.52827072143555 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.05859375 proc.cpu.threads‚17& disk.nvme0n1.out‚28609.24365234375 disk.drbd1716.in‚0 disk.rbd0.out‚0$ memory_percent‚94.75083351135254 cpu‚25.350966739579224 disk.sda.out‚ 7684.17578125 disk.sda2.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 378.98046875 disk.rbd1.out‚ 15.1396484375' proc.memory.availableMB‚ 860.0234375 network.sent‚ 1.9676277e+07 disk.drbd1716.out‚0 disk.drbd2266.in‚02¾·´’‡ ‚(:‡ Ö°ãÔ’죮 gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚129.179' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.8358 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 †ûxˆ ‚(:í Ö°ãÔõ°" disk.dm-0.out‚30112.9794921875 disk.nvme0n1.in‚ 379.54296875 disk.sda.in‚ 0.05859375 disk.drbd2265.in‚0 disk.drbd2080.out‚0 cpu‚25.35023905501181 network.sent‚ 1.9784311e+07' disk.nvme0c0n1.out‚30129.3544921875% disk.nvme0n1.out‚30129.3544921875 disk.sda.out‚ 7684.84375 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 network.recv‚ 2.263709e+06 disk.sda3.out‚ 7684.84375 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0$ memory_percent‚94.74923610687256# proc.memory.rssMB‚ 2603.19140625% disk./.usageGB‚234.18676376342773" disk.nvme0c0n1.in‚ 379.54296875 disk.dm-1.out‚ 7684.84375 disk.rbd0.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0* proc.memory.percent‚15.888619422912598 disk.dm-0.in‚ 379.54296875 disk.sda1.in‚0 disk.rbd1.out‚ 15.171875 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.in‚ 0.05859375 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.05859375( proc.memory.availableMB‚ 860.28515625* disk./.usagePercent‚15.716470211946385¬³Ÿ2‰ - _wandbruntime‚6383  _runtime‚6383+.ÁìÆŠ À _step‚662 & lr-AdamW/pg1‚6.380870148921338e-05 & lr-AdamW/pg2‚6.380870148921338e-05 trainer/global_step‚8099 $ _timestamp‚1.788401758552782e+09 _runtime‚6383.189606445C7|¼Ë‹ Å & lr-AdamW/pg1‚6.380870148921338e-05 & lr-AdamW/pg2‚6.380870148921338e-05 trainer/global_step‚8099 $ _timestamp‚1.788401758552782e+09 _runtime‚6383.189606445 _step‚662–0›@2Œ - _wandbruntime‚6383  _runtime‚6383x´vÉí' çO JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163638341 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108126704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106595803 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚228025557797888 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11418094801920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚253231850496000 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11615632997376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 163618303 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11418094801920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚544 % _timestamp‚1.7884017585534747e+09 _step‚663 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚216916424131584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 270214104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106595801 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 163618303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206390272 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11385363042952 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _runtime‚6383.190510691 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚239443634596864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163638341 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45907254 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚228025557797888 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚228025539794944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11418263801856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62218904 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11385197921548 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408ƒRÒL Ž ìO SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚239443634596864 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚253232080826368 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11385363042952 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _runtime‚6383.190510691 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚239443634596864 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 163638341 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚45907254 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batcp œ–¦h_start/active_bytes.large_pool.allocated‚228025557797888 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚228025539794944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11418263801856 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚62218904 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11385197921548 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 163638341 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108126704 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 106595803 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚228025557797888 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11418094801920 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚253231850496000 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11615632997376 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 163618303 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11418094801920 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚544 % _timestamp‚1.7884017585534747e+09 _step‚663 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚216916424131584 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 270214104 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 106595801 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 163618303 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 206390272 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 270266616 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 163658154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚253262180579840 _runtime‚6383.882242108 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62226283 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚253262415091200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 270246550 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108138905 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚216942367313920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45913159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11386752207704 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _step‚664 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 270266616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚239472484115968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11419487940096 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11419487940096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106608460 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚241645130446848 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11617260704256 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚239472484115968 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106608460 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 270246550 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163658154 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚228328954400012 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11419656953344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11419656953344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 163638090 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚228052827162624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚228052809159680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106608462 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11617050132992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108139442 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62226281 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45912624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 $ _timestamp‚1.788401759244948e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8099 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚228329136560984 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416Ô«‘û&‘ õM J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚216942384353280 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 106608462 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11419487940096 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11419487940096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 106608460 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚241645130446848 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11617260704256 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚239472484115968 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 106608460 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 270246550 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 163658154 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚228328954400012 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11419656953344 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11419656953344 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 163638090 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚228052827162624 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚228052809159680 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 106608462 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11617050132992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108139442 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62226281 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚45912624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 $ _timestamp‚1.788401759244948e+09 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8099 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚228329136560984 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚239472297099776 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚228052809159680 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚241645154386944 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 270266616 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 163658154 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚253262180579840 _runtime‚6383.882242108 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62226283 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚253262415091200 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 270246550 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108138905 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚216942367313920 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚45913159 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11386752207704 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _step‚664 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 270266616 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚239472484115968 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E9765625 disk.drbd2265.out‚0# proc.memory.rssMB‚ 2603.19140625 disk.rbd1.out‚15.23486328125 disk.sda2.in‚0 disk.dm-1.in‚ 0.05859375 disk.dm-1.out‚ 7686.66796875 disk.drbd2265.in‚0 disk.sda3.out‚ 7686.66796875* disk./.usagePercent‚15.724851642609877 network.sent‚ 1.9899158e+07•Ž°»2™ - _wandbruntime‚6419  _runtime‚6419„0ÇüÆš À _step‚666 & lr-AdamW/pg1‚6.239543662863513e-05 & lr-AdamW/pg2‚6.239543662863513e-05 trainer/global_step‚8149 $ _timestamp‚1.788401795029741e+09 _runtime‚6419.666614485Úˆ9Ë› Å & lr-AdamW/pg1‚6.239543662863513e-05 & lr-AdamW/pg2‚6.239543662863513e-05 trainer/global_step‚8149 $ _timestamp‚1.788401795029741e+09 _runtime‚6419.666614485 _step‚666š d2œ - _wandbruntime‚6419  _runtime‚6419Üîëí' çO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚229389008031744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164631043 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11487766124544 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11487935124480 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164611005 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108767375 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218213600287744 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚240876961159168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚501 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107228751 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚240876961159168 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11487766124544 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚229389026034688 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229389008031744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271859796 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11454669739956 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚218213583248384 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚240876774156288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚229668435149104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚240876774156288 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚243061888849920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107228753 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2xB%žò'ž ìO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚254748362594816 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46179619 trainer/global_step‚8149 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚254748597119488 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11454669739956 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚218213583248384 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚240876774156288 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚229668435149104 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚240876774156288 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚243061888849920 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107228753 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚229389008031744 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 164631043 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11487766124544 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11487935124480 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 164611005 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 108767375 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚218213600287744 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚240876961159168 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚501 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107228751 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚240876961159168 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11487766124544 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚229389026034688 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚229389008031744 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 271859796 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11688123942912 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚8149 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 271892268 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46184671 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107241410 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108779802 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62595133 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11489159262720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚243090777858048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11456058904500 % _timestamp‚1.7884017957044616e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 271892268 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62595131 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚240905623675392 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚229416295399424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚478 _step‚668 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚6420.341863312 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107241412 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164650856 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46185147 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11489159262720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108780280 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚240905436659200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164630792 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164650856 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚476 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11489328275968 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚229416295399424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 271872202 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚229416277396480 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L=DeviceÏ¢“•StatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972ö†Í6ü&£ öM OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11688123942912 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 trainer/global_step‚8149 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 271892268 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46184671 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107241410 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 108779802 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62595133 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11489159262720 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚243090777858048 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11456058904500 % _timestamp‚1.7884017957044616e+09 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 271892268 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62595131 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚240905623675392 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚229416295399424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚478 _step‚668 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚6420.341863312 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 107241412 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 164650856 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46185147 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11489159262720 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 108780280 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚240905436659200 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 164630792 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 164650856 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚476 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11489328275968 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚229416295399424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 271872202 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚229416277396480 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408œ¡œf2¤ - _wandbruntime‚6420  _runtime‚6420¹²N¾W¥ Ñ 0loss/FBt4/assignment_loss‚0.1296248472195448 0loss/SRqqt3/detection_loss‚1.714890573851087 .loss/FBt2/detection_loss‚2.391275316476822 _step‚669 2loss/SRqqt4/assignment_loss‚0.7869014143114527 /loss/FRt2/detection_loss‚3.1379149103164674 1loss/SRqqt1/detection_loss‚1.9442524840434392 /loss/FRt1/assignment_loss‚3.080403816699982 /loss/FRt1/detection_loss‚1.9751271667820214 .loss/FRt4/detection_loss‚1.667979496717453 1loss/SRqqt4/detection_loss‚1.0475958544914037 .loss/FBt1/detection_loss‚2.389318998903036 %loss/total_loss‚43.10321209261152 /loss/FRt3/assignment_loss‚3.514427391688029 2loss/SRqqt1/assignment_loss‚1.3505000274110999 0loss/FBt1/assignment_loss‚0.2955982572136122 1loss/SRqqt2/detection_loss‚2.1717578689257304 % _timestamp‚1.7884017957062087e+09 /loss/FRt2/assignment_loss‚3.090549430847168 2loss/SRqqt3/assignment_loss‚1.2827948954972355 0loss/FBt3/assignment_loss‚0.1736046633884598 _runtime‚6420.342939753 /loss/FBt4/detection_loss‚0.7106244212043458 epoch‚6 trainer/global_step‚8149 0loss/FRt4/assignment_loss‚1.4781624698638915 .loss/FBt3/detection_loss‚2.844326059023539 2loss/SRqqt2/assignment_loss‚2.1301334142684936 0loss/FBt2/assignment_loss‚0.6434871582314372 /loss/FRt3/detection_loss‚3.1519611552357674T}³T\¦ Ö .loss/FBt1/detection_loss‚2.389318998903036 %loss/total_loss‚43.10321209261152 /loss/FRt3/assignment_loss‚3.514427391688029 2loss/SRqqt1/assignment_loss‚1.3505000274110999 0loss/FBt1/assignment_loss‚0.2955982572136122 1loss/SRqqt2/detection_loss‚2.1717578689257304 % _timestamp‚1.7884017957062087e+09 /loss/FRt2/assignment_loss‚3.090549430847168 2loss/SRqqt3/assignment_loss‚1.2827948954972355 0loss/FBt3/assignment_loss‚0.1736046633884598 _runtime‚6420.342939753 /loss/FBt4/detection_loss‚0.7106244212043458 epoch‚6 trainer/global_step‚8149 0loss/FRt4/assignment_loss‚1.4781624698638915 .loss/FBt3/detection_loss‚2.844326059023539 2loss/SRqqt2/assignment_loss‚2.1301334142684936 0loss/FBt2/assignment_loss‚0.6434871582314372 /loss/FRt3/detection_loss‚3.1519611552357674 0loss/FBt4/assignment_loss‚0.1296248472195448 0loss/SRqqt3/detection_loss‚1.714890573851087 .loss/FBt2/detection_loss‚2.391275316476822 _step‚669 2loss/SRqqt4/assignment_loss‚0.7869014143114527 /loss/FRt2/detection_loss‚3.1379149103164674 1loss/SRqqt1/detection_loss‚1.9442524840434392 /loss/FRt1/assignment_loss‚3.080403816699982 /loss/FRt1/detection_loss‚1.9751271667820214 .loss/FRt4/detection_loss‚1.667979496717453 1loss/SRqqt4/detection_loss‚1.0475958544914037±ØÔ‹œ§ ‚(:‘ ’±ãԺꩮ gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚129.42' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.883999999999997 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ð1›u¨ ‚(:ê ’±ãÔ¿Ç°# memory_percent‚94.7488784790039# proc.memory.rssMB‚ 2603.19140625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 395.65234375 disk.dm-1.in‚0.0625 network.recv‚ 2.284166e+06% disk./.usageGB‚234.30936813354492% disk.nvme0n1.out‚33520.0771484375 disk.sda2.in‚0 disk.drbd2080.in‚0" disk.dm-0.out‚33520.0771484375' disk.nvme0c0n1.out‚33520.0771484375 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0* proc.memory.percent‚15.888619422912598 disk.drbd2266.out‚0 disk.dm-0.in‚ 395.65234375 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 7823.71484375 disk.sda.out‚ 7823.71484375 disk.dm-1.out‚ 7823.71484375 disk.drbd2080.out‚0 disk.rbd1.out‚ 15.2978515625 cpu‚25.347448035992038 network.sent‚ 2.000219e+07* disk./.usagePercent‚15.724698294096868 disk.sda.in‚0.0625 disk.sda2.out‚0% proc.memory.availableMB‚ 860.34375 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda3.in‚0.0625" disk.nvme0c0n1.in‚ 395.65234375`^‰#’© ‚(:‡ ¡±ãÔ¨•‚® gpu.0.gpu‚25.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚135.619' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.1238 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿIŸ„ª ‚(:ù ¡±ãÔ‘Â’± disk.sda.out‚ 7860.0078125 disk.sda2.in‚0 disk.dm-1.in‚ 0.06640625 cpu‚25.34675387167483& disk.nvme0n1.out‚33520.33642578125 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.sda3.out‚ 7860.0078125 disk.rbd1.in‚0 disk.sda.in‚ 0.06640625 disk.drbd2080.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 395.65234375# disk.dm-0.out‚33520.33642578125# proc.memory.rssMB‚ 2603.19140625 network.recv‚ 2.290241e+06 disk.sda3.in‚ 0.06640625 disk.nvme0n1.in‚ 395.65234375 disk.drbd1716.in‚0 disk.dm-1.out‚ 7860.0078125 disk.drbd2266.in‚0 proc.cpu.threads‚17' proc.memory.availableMB‚ 860.0859375 network.sent‚ 2.0100365e+07 disk.sda2.out‚0 disk.drbd1716.out‚0" disk.nvme0c0n1.in‚ 395.65234375 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 15.2978515625 disk.sda1.in‚0$ memory_percent‚94.75045204162598* proc.memory.percent‚15.888619422912598* disk./.usagePercent‚15.724698294096868% disk./.usageGB‚234.30936813354492( disk.nvme0c0n1.out‚33520.33642578125KÕ 2« - _wandbruntime‚6455  _runtime‚6455,Å1šÇ¬ Á & lr-AdamW/pg1‚6.099157009661044e-05 & lr-AdamW/pg2‚6.099157009661044e-05 trainer/global_step‚8199 % _timestamp‚1.7884018306658885e+09 _runtime‚6455.302722219 _step‚670‰™³Ì­ Æ & lr-AdamW/pg1‚6.099157009661044e-05 & lr-AdamW/pg2‚6.099157009661044e-05 trainer/global_step‚8199 % _timestamp‚1.7884018306658885e+09 _runtime‚6455.302722219 _step‚670ž×~2® - _wandbruntime‚6455  _runtime‚6455µ-ˆ‘í'¯ çO UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 165623745 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11557437447168 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚242310100718592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46451688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚219510742365184 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11557606447104 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107861703 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚502 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚256265089886208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 165603707 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11557606447104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62956606 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107861701 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚242310100718592 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 165603707 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚231034883923548 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46452190 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚256264859555840 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚230752494271488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242309913715712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚242309913715712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230330368 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107861703 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109408796 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11524306679768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 trainer/global_step‚8199 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚230752476268544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚231035066084312 _step‚671 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚230752494271488 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109408292 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚244507512321024 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 165623745 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11524141558364 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 % _timestamp‚1.7884018306665828e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚244507536261120 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11557437447168 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚242310100718592 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46451688 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚219510742365184 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11557606447104 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 107861703 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚502 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚256265089886208 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 165603707 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11557606447104 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚62956606 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 107861701 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚242310100718592 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 165603707 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚231034883923548 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46452190 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚256264859555840 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚230752494271488 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚242309913715712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚242309913715712 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 230330368 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 107861703 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 109408796 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11524306679768 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 trainer/global_step‚8199 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚230752476268544 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚231035066084312 _step‚671 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚230752494271488 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 109408292 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚244507512321024 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 165623745 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11524141558364 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 % _timestamp‚1.7884018306665828e+09 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚244507536261120 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165643558 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109421019 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚230779745633280 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62963983 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚242338763234816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109421522 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273517920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11558830585344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62963981 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚256295427128832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚244536449209344 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244536425269248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107874360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107874360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 273497854 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚230779763636224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 273497854 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11558999598592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚242338576218624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46457038 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165623494 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚230779745633280 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11525530722908 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107874362 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚231062398431400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚219536702586880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 165623494 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11558830585344 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46457539 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚219536685547520 _step‚672 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚242338763234816 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 165643558 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 109421019 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚230779745633280 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚62963983 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚242338763234816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 109421522 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 273517920 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11558830585344 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚62963981 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚256295427128832 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚244536449209344 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚244536425269248 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 107874360 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 107874360 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 273497854 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚230779763636224 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 273497854 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11558999598592 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚242338576218624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46457038 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 165623494 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚230779745633280 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11525530722908 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 107874362 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚231062398431400 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚219536702586880 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 165623494 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 Ótçƒ2´ - _wandbruntime‚6455  _runtime‚6455T« Tµ Î 0loss/FBt2/assignment_loss‚0.4118955990103643 /loss/FBt3/assignment_loss‚0.261917310681796 epoch‚6 0loss/FRt4/assignment_loss‚1.9230298860506578 2loss/SRqqt3/assignment_loss‚1.2717703706026078 0loss/SRqqt2/detection_loss‚2.334991529583931 0loss/FRt3/assignment_loss‚2.9074784288039575 2loss/SRqqt4/assignment_loss‚1.1483680063253963 0loss/FBt1/assignment_loss‚0.2565548109388024 &loss/total_loss‚42.834849640640314 0loss/FRt2/assignment_loss‚3.5095520416895547 0loss/FBt4/assignment_loss‚0.1801197538869201 /loss/FBt4/detection_loss‚0.7897420504689217 _runtime‚6455.977513502 trainer/global_step‚8199 1loss/SRqqt2/assignment_loss‚1.805099223789416 -loss/FRt1/detection_loss‚2.15208562190334 /loss/FRt2/detection_loss‚2.7679270108540854 /loss/FRt3/detection_loss‚2.3491473331665382 1loss/SRqqt3/detection_loss‚1.7114425422251225 /loss/FRt1/assignment_loss‚2.749018494288127 1loss/SRqqt1/detection_loss‚1.8969388982901971 0loss/SRqqt4/detection_loss‚1.022842156909295 .loss/FBt3/detection_loss‚2.710370513525876 .loss/FBt2/detection_loss‚2.680529158562422 _step‚673 1loss/SRqqt1/assignment_loss‚1.786287880463836 /loss/FRt4/detection_loss‚1.9788603220473637 -loss/FBt1/detection_loss‚2.22888069657179 % _timestamp‚1.7884018313407798e+09Õ’ËY¶ Ó 0loss/FRt4/assignment_loss‚1.9230298860506578 2loss/SRqqt3/assignment_loss‚1.2717703706026078 0loss/SRqqt2/detection_loss‚2.334991529583931 0loss/FRt3/assignment_loss‚2.9074784288039575 2loss/SRqqt4/assignment_loss‚1.1483680063253963 0loss/FBt1/assignment_loss‚0.2565548109388024 &loss/total_loss‚42.834849640640314 0loss/FRt2/assignment_loss‚3.5095520416895547 0loss/FBt4/assignment_loss‚0.1801197538869201 /loss/FBt4/detection_loss‚0.7897420504689217 _runtime‚6455.977513502 trainer/global_step‚8199 1loss/SRqqt2/assignment_loss‚1.805099223789416 -loss/FRt1/detection_loss‚2.15208562190334 /loss/FRt2/detection_loss‚2.7679270108540854 /loss/FRt3/detection_loss‚2.3491473331665382 1loss/SRqqt3/detection_loss‚1.7114425422251225 /loss/FRt1/assignment_loss‚2.749018494288127 1loss/SRqqt1/detection_loss‚1.8969388982901971 0loss/SRqqt4/detection_loss‚1.022842156909295 .loss/FBt3/detection_loss‚2.710370513525876 .loss/FBt2/detection_loss‚2.680529158562422 _step‚673 1loss/SRqqt1/assignment_loss‚1.786287880463836 /loss/FRt4/detection_loss‚1.9788603220473637 -loss/FBt1/detection_loss‚2.22888069657179 % _timestamp‚1.7884018313407798e+09 0loss/FBt2/assignment_loss‚0.4118955990103643 /loss/FBt3/assignment_loss‚0.261917310681796 epoch‚6¡»õžÉ· ‚(:’ °±ãÔ”ÿ¨® gpu.0.gpu‚15.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚146.131' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.226200000000002 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0þ´Úýq¸ ‚(:æ °±ãÔ“ৱ disk.rbd1.in‚0 disk.sda.in‚ 0.06640625 disk.sda.out‚ 7974.57421875 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.06640625 proc.cpu.threads‚17% disk./.usageGB‚234.30935668945312 disk.rbd0.out‚0 disk.sda3.out‚ 7974.57421875" disk.nvme0c0n1.in‚ 395.65234375 disk.nvme0n1.in‚ 395.65234375# proc.memory.rssMB‚ 2603.19140625 disk.drbd2265.out‚0 disk.sda3.in‚ 0.06640625 disk.nvme0n1.out‚ 33521.9375 cpu‚25.34606672079793 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 395.65234375 disk.sda1.out‚0! disk.nvme0c0n1.out‚ 33521.9375* proc.memory.percent‚15.888619422912598* disk./.usagePercent‚15.724697526074266 disk.dm-0.out‚ 33521.9375 disk.sda1.in‚0' proc.memory.availableMB‚ 859.8828125 network.sent‚ 2.0106263e+07 network.recv‚ 2.294345e+06 disk.drbd2266.in‚0 disk.dm-1.out‚ 7974.57421875# memory_percent‚94.7516918182373 disk.drbd2265.in‚0 disk.rbd1.out‚ 15.2978515625¬öX´s¹ ‚(:è ¿±ãÔ˜•± disk.nvme0n1.in‚ 395.65234375 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd1716.out‚0 cpu‚25.345385079791416 disk.drbd2080.out‚0$ disk./.usageGB‚234.3094139099121 disk.sda3.out‚ 7975.03125 disk.rbd1.out‚ 15.2978515625# disk.nvme0n1.out‚33522.10546875 disk.sda2.in‚0 disk.sda1.out‚0# proc.memory.rssMB‚ 2603.19140625 network.sent‚ 2.0215275e+07) disk./.usagePercent‚15.72470136618728" disk.nvme0c0n1.in‚ 395.65234375 disk.rbd0.out‚0 disk.dm-0.in‚ 395.65234375& proc.memory.availableMB‚ 860.140625* proc.memory.percent‚15.888619422912598 proc.cpu.threads‚17 disk.dm-1.out‚ 7975.03125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚ 0.06640625 disk.rbd1.in‚0 disk.dm-1.in‚ 0.06640625 disk.rbd0.in‚0 disk.sda2.out‚0% disk.nvme0c0n1.out‚33522.10546875 disk.sda.in‚ 0.06640625 disk.sda.out‚ 7975.03125 disk.dm-0.out‚33522.10546875 disk.sda1.in‚0$ memory_percent‚94.75011825561523 network.recv‚ 2.301063e+06 disk.drbd2266.out‚0Š”^ª’º ‚(:‡ ¿±ãÔÆ°ºÉ gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚115.553' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1106 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0·µTê2» - _wandbruntime‚6491  _runtime‚6491ðØ”vɼ à ' lr-AdamW/pg1‚5.9597394456820464e-05 ' lr-AdamW/pg2‚5.9597394456820464e-05 trainer/global_step‚8249 % _timestamp‚1.7884018666829767e+09 _runtime‚6491.319820212 _step‚674g4 ν È ' lr-AdamW/pg1‚5.9597394456820464e-05 ' lr-AdamW/pg2‚5.9597394456820464e-05 trainer/global_step‚8249 % _timestamp‚1.7884018666829767e+09 _runtime‚6491.319820212 _step‚674¢+,ÿ52¾ - _wandbruntime‚6491  _runtime‚6491‡•žl¢¿ çO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63325454 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚675 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108494653 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚220807918521344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚232115962508288 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110048513 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LFDeviceSta¶_~LK tsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11627277769728 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11593613376772 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11627108769792 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚232115944505344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 166616447 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46722520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11828426174976 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11828213493248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚232401697019520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 % _timestamp‚1.7884018666836612e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108494651 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 275091060 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166616447 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚232401514858756 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚243743053275136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚243743240278016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚245953183672320 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚245953159732224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1?¿¬–ò'À ìO % _timestamp‚1.7884018666836612e+09 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 108494651 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 275091060 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 166616447 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚232401514858756 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚243743053275136 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚243743240278016 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚245953183672320 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚245953159732224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63325454 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚675 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 108494653 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚220807918521344 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚232115962508288 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 110048513 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11627277769728 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11593613376772 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11627108769792 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚232115944505344 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 166616447 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46722520 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11828426174976 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11828213493248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚232401697019520 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0£?™¯ó2Á - _wandbruntime‚6491  _runtime‚6491–‡„÷& ñM VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 166616196 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11595002541316 _step‚676 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108507310 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7884018673602204e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11829626198016 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚243771715778048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110060720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11628501907968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46727889 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11595167662928 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275143572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚243771902794240 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 275123506 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46728409 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚220833844664320 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11829840963584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110061242 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108507310 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 166636260 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11628670921216 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11628501907968 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108507312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11628670921216 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚245982096620544 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166616196 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚257811937584128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚257811698878464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _runtime‚6491.997513478 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 275143572 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚220833861703680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248-áwà öM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110060720 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11628501907968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚46727889 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11595167662928 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 275143572 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚243771902794240 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 275123506 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚46728409 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚220833844664320 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11829840963584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/i­¡*…nactive_split.all.allocated‚ 110061242 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 108507310 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 166636260 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11628670921216 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11628501907968 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 108507312 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11628670921216 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚245982096620544 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 166616196 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚257811937584128 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚257811698878464 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _runtime‚6491.997513478 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 275143572 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚220833861703680 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 166616196 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11595002541316 _step‚676 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 108507310 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 % _timestamp‚1.7884018673602204e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11829626198016 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚243771715778048 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚233479430745088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46995073 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 167589111 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109127601 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚222105060598784 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚233479412742144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46994524 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚245176192834560 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11696949092352 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109127603 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63694306 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218973184 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11663085195180 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167589111 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276736752 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11696780092416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110688828 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚259297903484928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚247398831083520 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚233479430745088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚233479412742144 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11696949092352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109127603 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚222105077638144 $ _timestamp‚1.788401902497299e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚245176379837440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚247398807143424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚259298146398208 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11899315314688 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚245176192834560 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚6527.134374846 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚245176379837440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63694304 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚233768145793964 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11696780092416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚233479430745088 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚551 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚46995073 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 167589111 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 109127601 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚222105060598784 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚233479412742144 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚46994524 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚245176192834560 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11696949092352 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109127603 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚63694306 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218973184 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11663085195180 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 167589111 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 276736752 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11696780092416 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 110688828 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚549 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚259297903484928 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚247398831083520 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚233479430745088 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚233479412742144 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11696949092352 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 109127603 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚222105077638144 $ _timestamp‚1.788401902497299e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚245176379837440 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚247398807143424 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚259298146398208 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚11899315314688 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚245176192834560 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚6527.134374846 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚245176379837440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚63694304 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚233768145793964 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11696780092416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064§lych2Ó - _wandbruntime‚6527  _runtime‚6527YÈÄO÷&Ô ñM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚233506682106880 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276769224 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚259328235235840 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 276769224 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109140262 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11698342243840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚222131020820480 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚233506700109824 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63701683 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167628962 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109140260 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47000175 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167608898 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11698342243840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚233506682106880 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚247427720091648 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11698173230592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47000709 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚259328469747200 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11900515144192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚233795660301816 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 276749158 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109140260 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110701856 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚222131003781120 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚233506700109824 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 167608898 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚233795478140844 _runtime‚6527.811803627 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11664474359724 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11900725715456 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110702392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11664639481336 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚245205042353664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 276749158 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109140260 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 110701856 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚222131003781120 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚233506700109824 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 167608898 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚233795478140844 _runtime‚6527.811803627 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11664474359724 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11900725715456 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 110702392 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11664639481336 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚245205042353664 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 276769224 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚259328235235840 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 276769224 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109140262 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11698342243840 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚222131020820480 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚233506700109824 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚63701683 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 167628962 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109140260 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47000175 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚534 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 167608898 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11698342243840 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚233506682106880 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚247427720091648 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11698173230592 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47000709 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚259328469747200 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚11900515144192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0¨ðàÈ·2Ö - _wandbruntime‚6527  _runtime‚6527Éîã]× × 0loss/FRt4/assignment_loss‚1.6415094027393742 .loss/FRt1/detection_loss‚2.349130573700059 trainer/global_step‚8299 _step‚681 1loss/FBt3/assignment_loss‚0.23987280577419767 1loss/FBt4/assignment_loss‚0.06680568199138648 1loss/SRqqt1/detection_loss‚1.8831202742126254 /loss/FBt2/detection_loss‚2.6768091718355813 _runtime‚6527.812846565 0loss/FRt2/assignment_loss‚3.7351097699367637 1loss/FBt2/assignment_loss‚0.14217068101519223 0loss/SRqqt3/detection_loss‚1.799803087297751 0loss/FRt1/assignment_loss‚3.2669328462402776 1loss/FBt1/assignment_loss‚0.15068364466421968 /loss/SRqqt2/detection_loss‚2.06944939494133 0loss/SRqqt4/detection_loss‚1.117019613758779 /loss/FBt1/detection_loss‚2.4431063532829285 % _timestamp‚1.7884019031760643e+09 1loss/SRqqt3/assignment_loss‚1.270790835348319 /loss/FRt4/detection_loss‚1.6826385284136784 epoch‚6 /loss/FBt4/detection_loss‚0.7543391465571605 &loss/total_loss‚43.647716589976454 2loss/SRqqt2/assignment_loss‚2.3639435186272575 /loss/FRt2/detection_loss‚2.8359008413372617 /loss/FBt3/detection_loss‚3.2313352823257446 0loss/FRt3/assignment_loss‚2.8198368472437703 2loss/SRqqt1/assignment_loss‚1.1934627200166383 2loss/SRqqt4/assignment_loss‚1.0148486913719648 .loss/FRt3/detection_loss‚2.899096877344193:õ,¶bØ Ü 0loss/FRt4/assignment_loss‚1.6415094027393742 .loss/FRt1/detection_loss‚2.349130573700059 trainer/global_step‚8299 _step‚681 1loss/FBt3/assignment_loss‚0.23987280577419767 1loss/FBt4/assignment_loss‚0.06680568199138648 1loss/SRqqt1/detection_loss‚1.8831202742126254 /loss/FBt2/detection_loss‚2.6768091718355813 _runtime‚6527.812846565 0loss/FRt2/assignment_loss‚3.7351097699367637 1loss/FBt2/assignment_loss‚0.14217068101519223 0loss/SRqqt3/detection_loss‚1.799803087297751 0loss/FRt1/assignment_loss‚3.2669328462402776 1loss/FBt1/assignment_loss‚0.15068364466421968 /loss/SRqqt2/detection_loss‚2.06944939494133 0loss/SRqqt4/detection_loss‚1.117019613758779 /loss/FBt1/detection_loss‚2.4431063532829285 % _timestamp‚1.7884019031760643e+09 1loss/SRqqt3/assignment_loss‚1.270790835348319 /loss/FRt4/detection_loss‚1.6826385284136784 epoch‚6 /loss/FBt4/detection_loss‚0.7543391465571605 &loss/total_loss‚43.647716589976454 2loss/SRqqt2/assignment_loss‚2.3639435186272575 /loss/FRt2/detection_loss‚2.8359008413372617 /loss/FBt3/detection_loss‚3.2313352823257446 0loss/FRt3/assignment_loss‚2.8198368472437703 2loss/SRqqt1/assignment_loss‚1.1934627200166383 2loss/SRqqt4/assignment_loss‚1.0148486913719648 .loss/FRt3/detection_loss‚2.899096877344193©óÆòð’Ù ‚(:‡ û±ãÔÛçú­ gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚131.601' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.3202 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0î¹pÅ~Ú ‚(:ó û±ãÔ¥Éá° disk.rbd1.out‚ 15.2978515625( disk.nvme0c0n1.out‚33523.32177734375* proc.memory.percent‚15.888619422912598 cpu‚25.342612549793323 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.sda3.in‚ 0.06640625& disk.nvme0n1.out‚33523.32177734375 disk.sda3.out‚ 7977.140625 disk.rbd1.in‚0 disk.dm-1.out‚ 7977.140625 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 395.65234375 disk.sda.out‚ 7977.140625 disk.drbd2266.out‚0 disk.dm-0.in‚ 395.65234375 network.recv‚ 2.3194e+06 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.06640625 proc.cpu.threads‚17 disk.sda.in‚ 0.06640625 disk.rbd0.in‚0 disk.rbd0.out‚0# proc.memory.rssMB‚ 2603.19140625 disk.drbd2266.in‚0* disk./.usagePercent‚15.724698550104401 disk.drbd2080.in‚0# disk.dm-0.out‚33523.32177734375 disk.nvme0n1.in‚ 395.65234375' proc.memory.availableMB‚ 859.5390625 disk.sda2.in‚0# memory_percent‚94.7537899017334 network.sent‚ 2.0341896e+07$ disk./.usageGB‚234.3093719482422 disk.drbd2265.out‚0zØqÛ ‚(:’ Š²ãÔ¢Œà¯ gpu.0.gpu‚34.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚134.552' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.910400000000003 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¿ÌG€Ü ‚(:õ Š²ãÔ”ú”°& disk.nvme0n1.out‚33524.41455078125 disk.drbd1716.out‚0 disk.dm-1.out‚ 7977.640625 disk.drbd2266.out‚0 disk.drbd2080.in‚0) disk./.usagePercent‚15.72469419797632" disk.nvme0c0n1.in‚ 395.65234375( disk.nvme0c0n1.out‚33524.41455078125 disk.sda.in‚ 0.06640625 disk.sda1.in‚0 disk.sda2.out‚0 disk.drbd2080.out‚0 network.sent‚ 2.0450009e+07 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2265.in‚0 disk.sda3.out‚ 7977.640625 disk.rbd1.in‚0 disk.nvme0n1.in‚ 395.65234375 disk.sda.out‚ 7977.640625 disk.drbd2266.in‚0 disk.rbd0.in‚0# proc.memory.rssMB‚ 2603.19140625 cpu‚25.341950306600285 network.recv‚ 2.324857e+06# disk.dm-0.out‚33524.41455078125 disk.sda3.in‚ 0.06640625$ memory_percent‚94.75393295288086* proc.memory.percent‚15.888619422912598 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.dm-0.in‚ 395.65234375 disk.rbd1.out‚ 15.2978515625& proc.memory.availableMB‚ 859.515625 proc.cpu.threads‚17% disk./.usageGB‚234.30930709838867 disk.rbd0.out‚0 disk.sda1.out‚0e£c2Ý - _wandbruntime‚6563  _runtime‚6563žEªðÇÞ Á % _timestamp‚1.7884019385349169e+09 _runtime‚6563.171761645 _step‚682 & lr-AdamW/pg1‚5.683927595028303e-05 & lr-AdamW/pg2‚5.683927595028303e-05 trainer/global_step‚8349$üY,Ìß Æ & lr-AdamW/pg2‚5.683927595028303e-05 trainer/global_step‚8349 % _timestamp‚1.7884019385349169e+09 _runtime‚6563.171761645 _step‚682 & lr-AdamW/pg1‚5.683927595028303e-05ª!‚^o2à - _wandbruntime‚6563  _runtime‚6563ˆ}ÞJá çO trainer/global_step‚8349 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚260814402404352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11766620414976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109760553 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚246609519396864 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 111327428 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚234842898981888 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚248844478494720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 168601851 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246609332393984 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚248844454554624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408oäøò'â ìO UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 168601851 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _runtime‚6563.172669718 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 168581813 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 278362404 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚11969947849728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚223402219715584 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚260814636929024 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚11732722134992 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚235134776729172 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _step‚683 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64063154 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 % _timestamp‚1.7884019385356064e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 109760551 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11766451415040 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚248844454554624 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 trainer/global_step‚8349 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚260814402404352 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11766620414976 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 109760553 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚246609519396864 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 111327428 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚234842898981888 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚248844478494720 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 168601851 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚246609332393984 J6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 278374810 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111340778 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 278374810 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚246638181913088 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111340226 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚248873367502848 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚235162109076052 trainer/global_step‚8349 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚248873391442944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47270245 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚246638181913088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 168601600 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚234870150343680 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚260844729504256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11734111299744 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11767844553216 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109773210 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11768013566464 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109773210 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109773212 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 278394876 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47269695 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11971576766976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 % _timestamp‚1.7884019392073846e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚550 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11768013566464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚260844968209920 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _runtime‚6563.844632745 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 168601600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚234870150343680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64070531 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚235162291237024¿¬ Ëå öM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 278394876 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 168621664 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚552 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11767844553216 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 278394876 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚47269695 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚11971576766976 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 % _timestamp‚1.7884019392073846e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚550 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11768013566464 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚260844968209920 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _runtime‚6563.844632745 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 168601600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚234870150343680 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64070531 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚235162291237024 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚246637994896896 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 278374810 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111340778 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 278374810 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚246638181913088 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111340226 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚248873367502848 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚235162109076052 trainer/global_step‚8349 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚248873391442944 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47270245 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚246638181913088 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 168601600 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UHDeviceStatsMonitor.on_train_batch_end/r#<âÝ equested_bytes.large_pool.current‚17039360 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚234870150343680 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚260844729504256 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11734111299744 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11767844553216 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 109773210 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11768013566464 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 109773210 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 109773212 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚250290101965824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110393503 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169594553 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚250290125905920 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚224699378832384 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚236206349215744 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11836291737600 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚262331153612800 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚262330914893824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚248042471953408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12040812928000 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚248042471953408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚236206367218688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 _runtime‚6598.992431973 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111968208 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169574515 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11836122737664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 279968016 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚248042658956288 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 279988056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 trainer/global_step‚8399 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64432004 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47536202 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110393501 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11836122737664 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚236206367218688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47535677 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416Í0 6ð'ò êO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 110393503 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 169594553 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚250290125905920 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚224699378832384 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚236206349215744 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11836291737600 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚262331153612800 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚262330914893824 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚248042471953408 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12040812928000 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚248042471953408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚236206367218688 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 _runtime‚6598.992431973 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 111968208 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 169574515 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11836122737664 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 279968016 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚248042658956288 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 279988056 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 trainer/global_step‚8399 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64432004 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚47536202 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 110393501 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11836122737664 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚236206367218688 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47535677 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 # _timestamp‚1.78840197435534e+09 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚250290101965824 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0¯éó5Á2ó - _wandbruntime‚6599  _runtime‚6599Ê㨠ô ñM WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚236233618580480 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169614366 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11837515875840 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚236233636583424 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚528 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 169614366 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚250319038854144 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 111981025 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11837684889088 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚236233618580480 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚236528922172232 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚250319014914048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11837515875840 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884019750268564e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64439381 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚248071321472512 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚236233636583424 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚262361480074752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12042230649344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110406162 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚248071321472512 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚248071134456320 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚236528740011260 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11837684889088 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111980497 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11803583118152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _runtime‚6599.664243983 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110406162 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚224725339054080 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280000462 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110406160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280020528 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8399 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169594302 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280000462 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚224725322014720 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110406160 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚262361245563392 _step‚688 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169594302 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64439383 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚250319014914048 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11837515875840 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 % _timestamp‚1.7884019750268564e+09 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚64439381 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚248071321472512 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚236233636583424 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚262361480074752 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12042230649344 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 110406162 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚248071321472512 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚248071134456320 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚236528740011260 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11837684889088 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 111980497 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11803583118152 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _runtime‚6599.664243983 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 110406162 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚224725339054080 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 280000462 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 110406160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 280020528 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8399 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 169594302 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 280000462 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚224725322014720 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 110406160 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚262361245563392 _step‚688 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 169594302 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚64439383 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 169614366 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066°ujž2ö - _wandbruntime‚6599  _runtime‚6599VO‡¡V÷ Ð 0loss/FBt2/assignment_loss‚0.1477178273101624 0loss/FBt3/assignment_loss‚0.4483362456667237 .loss/FRt3/detection_loss‚2.122188280026118 /loss/FBt1/detection_loss‚2.4200110882520676 .loss/FBt2/detection_loss‚6.341277877489726 /loss/FRt2/assignment_loss‚4.149085217052036 1loss/SRqqt3/detection_loss‚1.6366179356487787 trainer/global_step‚8399 2loss/SRqqt3/assignment_loss‚1.5879853733065652 1loss/SRqqt4/assignment_loss‚0.990908433802617 1loss/FBt4/assignment_loss‚0.11270428696249231 .loss/SRqqt2/detection_loss‚2.2251385031268 /loss/FBt4/detection_loss‚0.7298585988488049 0loss/FRt1/assignment_loss‚2.6479967708857552 /loss/FBt3/detection_loss‚2.0550050884485245 epoch‚6 -loss/FRt2/detection_loss‚2.75803060664071 /loss/FRt4/detection_loss‚1.6348204837512161 1loss/SRqqt1/detection_loss‚1.6710473492424538 1loss/SRqqt4/detection_loss‚1.0621317469347746 1loss/SRqqt2/assignment_loss‚2.130865514976904 %loss/total_loss‚45.60001990938748 % _timestamp‚1.7884019750285122e+09 _runtime‚6599.665255782 1loss/FBt1/assignment_loss‚0.34292578002623486 _step‚689 2loss/SRqqt1/assignment_loss‚1.4740164782851934 /loss/FRt1/detection_loss‚2.0414274177461302 0loss/FRt3/assignment_loss‚3.5530845172264995 /loss/FRt4/assignment_loss‚1.3168384877301882UaR[ø Õ 0loss/FBt3/assignment_loss‚0.4483362456667237 .loss/FRt3/detection_loss‚2.122188280026118 /loss/FBt1/detection_loss‚2.4200110882520676 .loss/FBt2/detection_loss‚6.341277877489726 /loss/FRt2/assignment_loss‚4.149085217052036 1loss/SRqqt3/detection_loss‚1.6366179356487787 trainer/global_step‚8399 2loss/SRqqt3/assignment_loss‚1.5879853733065652 1loss/SRqqt4/assignment_loss‚0.990908433802617 1loss/FBt4/assignment_loss‚0.11270428696249231 .loss/SRqqt2/detection_loss‚2.2251385031268 /loss/FBt4/detection_loss‚0.7298585988488049 0loss/FRt1/assignment_loss‚2.6479967708857552 /loss/FBt3/detection_loss‚2.0550050884485245 epoch‚6 -loss/FRt2/detection_loss‚2.75803060664071 /loss/FRt4/detection_loss‚1.6348204837512161 1loss/SRqqt1/detection_loss‚1.6710473492424538 1loss/SRqqt4/detection_loss‚1.0621317469347746 1loss/SRqqt2/assignment_loss‚2.130865514976904 %loss/total_loss‚45.60001990938748 % _timestamp‚1.7884019750285122e+09 _runtime‚6599.665255782 1loss/FBt1/assignment_loss‚0.34292578002623486 _step‚689 2loss/SRqqt1/assignment_loss‚1.4740164782851934 /loss/FRt1/detection_loss‚2.0414274177461302 0loss/FRt3/assignment_loss‚3.5530845172264995 /loss/FRt4/assignment_loss‚1.316838487730188 0loss/FBt2/assignment_loss‚0.1477178273101624±ˆfcÔù ‚(:… ·²ãÔåÓ ® gpu.0.gpu‚22.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚146.42' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.284 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«déÉ{ú ‚(:ð ·²ãÔ«òò±# proc.memory.rssMB‚ 2603.19140625 cpu‚25.33991569920199 network.recv‚ 2.340572e+06 disk.drbd2266.in‚0 disk.rbd1.in‚0* proc.memory.percent‚15.888619422912598 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 7979.43359375$ disk./.usageGB‚234.3091278076172 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.dm-0.in‚ 395.65234375# memory_percent‚94.7542667388916' proc.memory.availableMB‚ 859.4609375 network.sent‚ 2.0571042e+07 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.06640625 disk.sda3.in‚ 0.06640625# disk.nvme0n1.out‚33525.10546875 disk.sda2.out‚0 disk.drbd2080.out‚0 disk.dm-0.out‚33525.10546875 disk.rbd1.out‚ 15.2978515625 disk.sda.in‚ 0.06640625 disk.dm-1.out‚ 7979.43359375 disk.sda1.in‚0% disk.nvme0c0n1.out‚33525.10546875) disk./.usagePercent‚15.72468216562221 disk.rbd0.in‚0 proc.cpu.threads‚17 disk.nvme0n1.in‚ 395.65234375 disk.sda.out‚ 7979.43359375 disk.drbd2266.out‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 395.65234375Œö þœû ‚(:‘ ƲãÔÍЭ® gpu.0.gpu‚4.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚136.974' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.394799999999996 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Šãú¯|ü ‚(:ñ ƲãÔÒž¡° cpu‚25.3391984363991& disk.nvme0n1.out‚33526.67138671875 disk.sda2.out‚0" disk.nvme0c0n1.in‚ 395.65234375( disk.nvme0c0n1.out‚33526.67138671875 disk.nvme0n1.in‚ 395.65234375 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17$ memory_percent‚94.74983215332031% disk./.usageGB‚234.30805587768555 disk.rbd1.in‚0 disk.sda.in‚ 0.06640625 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 395.65234375 network.recv‚ 2.346147e+06 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 7983.546875 disk.rbd1.out‚ 15.2978515625 disk.sda2.in‚0 disk.dm-1.in‚ 0.06640625 disk.dm-1.out‚ 7983.546875 disk.sda1.out‚0 disk.sda3.in‚ 0.06640625 disk.sda.out‚ 7983.546875 disk.drbd2266.in‚0# disk.dm-0.out‚33526.67138671875$ proc.memory.availableMB‚860.1875# proc.memory.rssMB‚ 2604.47265625 network.sent‚ 2.0679253e+07) disk./.usagePercent‚15.72461022750509 disk.drbd2265.in‚0* proc.memory.percent‚15.896439552307129[ÌK’ý ‚(:‡ Õ²ãÔŠá”® gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚141.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.3634 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“º~Ý~þ ‚(:ó Õ²ãÔæÕ± disk.sda2.in‚0 disk.drbd1716.in‚0! disk.dm-0.out‚33531.427734375 disk.sda3.in‚ 0.06640625% disk./.usageGB‚234.30805587768555 disk.sda.out‚ 7985.27734375 disk.drbd2265.in‚0 disk.dm-0.in‚ 395.65234375 disk.sda1.out‚0# proc.memory.rssMB‚ 2604.47265625 disk.sda1.in‚0 network.sent‚ 2.0684573e+07 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.rbd1.in‚0$ disk.nvme0n1.out‚33531.427734375$ memory_percent‚94.74983215332031 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda3.out‚ 7985.27734375" disk.nvme0c0n1.in‚ 395.65234375 network.recv‚ 2.350271e+06) disk./.usagePercent‚15.72461022750509 disk.drbd2266.in‚0 disk.rbd1.out‚ 15.2978515625 cpu‚25.338535514180563 disk.sda.in‚ 0.06640625 disk.sda2.out‚0 disk.dm-1.in‚ 0.06640625 disk.rbd0.in‚0 disk.rbd0.out‚0$ proc.memory.availableMB‚860.1875 disk.drbd2266.out‚0 disk.dm-1.out‚ 7985.27734375& disk.nvme0c0n1.out‚33531.427734375 disk.nvme0n1.in‚ 395.65234375* proc.memory.percent‚15.896439552307129ð¡&ø2ÿ - _wandbruntime‚6634  _runtime‚6634ê‚iÇ€!Á % _timestamp‚1.7884020102302449e+09 _runtime‚6634.867069066 _step‚690 & lr-AdamW/pg1‚5.412338014028904e-05 & lr-AdamW/pg2‚5.412338014028904e-05 trainer/global_step‚8449æ;mÌ!Æ % _timestamp‚1.7884020102302449e+09 _runtime‚6634.867069066 _step‚690 & lr-AdamW/pg1‚5.412338014028904e-05 & lr-AdamW/pg2‚5.412338014028904e-05 trainer/global_step‚8449²_ Æ2‚!- _wandbruntime‚6634  _runtime‚6634‰T—ë8ƒ!çO ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 111026453 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚47808121 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚237569835455488 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 170587255 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚249475611512832 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11905794060288 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111026451 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚249475611512832 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚237569817452544 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12111910790656 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚237569835455488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11905963060224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚251735749377024 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12111698108928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚691 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 281593668 % _timestamp‚1.7884020102309163e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚225996537949184 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111026453 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11905794060288 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 112609525 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 112608975 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64800854 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 170587255 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚249475798515712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚249475798515712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚263847447485952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚237868220760352 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚225996554988544 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 trainer/global_step‚8449 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11905963060224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281613708 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚263847684107776 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚237868038599588 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11871500650404 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚249475611512832 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚11905794060288 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 111026451 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚550 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚249475611512832 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚237569817452544 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12111910790656 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚237569835455488 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚11905963060224 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚251735749377024 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12111698108928 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _step‚691 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 281593668 % _timestamp‚1.7884020102309163e+09 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚225996537949184 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 111026453 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11905794060288 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 112609525 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 112608975 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚64800854 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 170587255 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚249475798515712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚249475798515712 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚263847447485952 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚237868220760352 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚225996554988544 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 trainer/global_step‚8449 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚11905963060224 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 281613708 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚263847684107776 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚237868038599588 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11871500650404 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 112622148 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚251764686265344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170587004 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11907356211712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170607068 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11873054936560 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚249504461031936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚517 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 170587004 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112621631 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚237597086817280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11907187198464 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 $ _timestamp‚1.788402010911739e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚249504274015744 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11907356211712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚237597104820224 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚237597104820224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚226022481131520 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281626114 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47813915 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚263878014913536 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12113328648192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚249504461031936 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚226022498170880 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12113113882624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _runtime‚6635.549109629 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111039112 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111039110 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111039110 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 _step‚692 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281646180 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170607068 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0L1êE&‡!õM L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 112622148 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚251764686265344 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 170587004 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚11907356211712 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 170607068 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11873054936560 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚249504461031936 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚517 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 170587004 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 112621631 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚237597086817280 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚11907187198464 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 $ _timestamp‚1.788402010911739e+09 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚249504274015744 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚11907356211712 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚237597104820224 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚237597104820224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚226022481131520 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 281626114 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚47813915 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚263878014913536 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12113328648192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚249504461031936 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚226022498170880 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12113113882624 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _runtime‚6635.549109629 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 111039112 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 111039110 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 111039110 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 _step‚692 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 281646180 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 170607068 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ZFDeviceStats»GõíMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚237597086817280 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408´öž@°2ˆ!- _wandbruntime‚6635  _runtime‚6635Œ*ÂÃW‰!Ñ /loss/FRt1/detection_loss‚2.1247353275616963 .loss/FRt2/detection_loss‚2.624298427786146 1loss/SRqqt2/detection_loss‚2.5987693558136624 1loss/SRqqt3/detection_loss‚1.7720755311846732 1loss/SRqqt4/detection_loss‚1.0665384804880298 .loss/FBt1/detection_loss‚2.841585159301758 1loss/SRqqt1/assignment_loss‚1.634360586895662 /loss/FRt4/detection_loss‚1.9093474491554148 0loss/FRt3/assignment_loss‚2.3399351757505666 0loss/FRt4/assignment_loss‚1.4050243228089576 _runtime‚6635.550114935 .loss/FRt1/assignment_loss‚3.32687287595537 2loss/SRqqt2/assignment_loss‚1.8426624943812688 1loss/SRqqt3/assignment_loss‚1.542394033074379 .loss/FBt2/detection_loss‚3.081368829522814 .loss/FBt3/detection_loss‚2.487209665775299 %loss/total_loss‚43.57139838258719 1loss/SRqqt1/detection_loss‚1.4827391626861166 epoch‚6 .loss/FRt2/assignment_loss‚4.31993750163487 0loss/FBt3/assignment_loss‚0.5090458921563368 .loss/FRt3/detection_loss‚2.249277981079143 % _timestamp‚1.7884020109133732e+09 2loss/SRqqt4/assignment_loss‚1.0181782448135719 1loss/FBt1/assignment_loss‚0.10500514527250614 /loss/FBt4/detection_loss‚0.8023778868639576 trainer/global_step‚8449 1loss/FBt2/assignment_loss‚0.32346815216754166 1loss/FBt4/assignment_loss‚0.16419070045744938 _step‚693® È\Š!Ö 0loss/FRt3/assignment_loss‚2.3399351757505666 0loss/FRt4/assignment_loss‚1.4050243228089576 _runtime‚6635.550114935 .loss/FRt1/assignment_loss‚3.32687287595537 2loss/SRqqt2/assignment_loss‚1.8426624943812688 1loss/SRqqt3/assignment_loss‚1.542394033074379 .loss/FBt2/detection_loss‚3.081368829522814 .loss/FBt3/detection_loss‚2.487209665775299 %loss/total_loss‚43.57139838258719 1loss/SRqqt1/detection_loss‚1.4827391626861166 epoch‚6 .loss/FRt2/assignment_loss‚4.31993750163487 0loss/FBt3/assignment_loss‚0.5090458921563368 .loss/FRt3/detection_loss‚2.249277981079143 % _timestamp‚1.7884020109133732e+09 2loss/SRqqt4/assignment_loss‚1.0181782448135719 1loss/FBt1/assignment_loss‚0.10500514527250614 /loss/FBt4/detection_loss‚0.8023778868639576 trainer/global_step‚8449 1loss/FBt2/assignment_loss‚0.32346815216754166 1loss/FBt4/assignment_loss‚0.16419070045744938 _step‚693 /loss/FRt1/detection_loss‚2.1247353275616963 .loss/FRt2/detection_loss‚2.624298427786146 1loss/SRqqt2/detection_loss‚2.5987693558136624 1loss/SRqqt3/detection_loss‚1.7720755311846732 1loss/SRqqt4/detection_loss‚1.0665384804880298 .loss/FBt1/detection_loss‚2.841585159301758 1loss/SRqqt1/assignment_loss‚1.634360586895662 /loss/FRt4/detection_loss‚1.9093474491554148µ{š“’‹!‚(:‡ ä²ãÔŒ—÷­ gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚144.824' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.9648 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0>°÷c„Œ!‚(:ù ä²ãÔÏøŽ°* proc.memory.percent‚15.896439552307129 disk.rbd1.in‚0& disk.nvme0n1.out‚33535.84326171875 disk.sda.in‚ 0.06640625 disk.sda3.in‚ 0.06640625 disk.sda3.out‚ 7986.8984375 disk.dm-1.in‚ 0.06640625$ memory_percent‚94.75007057189941# disk.dm-0.out‚33535.84326171875 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 7986.8984375 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚33535.84326171875 disk.nvme0n1.in‚ 395.65234375 disk.sda.out‚ 7986.8984375 disk.drbd1716.out‚0 disk.sda1.in‚0% disk./.usageGB‚232.17800521850586 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0' proc.memory.availableMB‚ 860.1484375 network.sent‚ 2.0690411e+07 network.recv‚ 2.354262e+06* disk./.usagePercent‚15.581660740535996 disk.drbd1716.in‚0# proc.memory.rssMB‚ 2604.47265625 disk.dm-0.in‚ 395.65234375 disk.rbd1.out‚ 15.2978515625 disk.sda2.in‚0 cpu‚25.33786135547206 proc.cpu.threads‚17 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 395.65234375 disk.drbd2266.in‚0–¿â<’!‚(:‡ ó²ãԷο¯ gpu.0.gpu‚43.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚122.154' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“ÿ·ƒŽ!‚(:ø ó²ãÔæÍë° network.sent‚ 2.0799382e+07 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.sda3.out‚ 7987.4765625 disk.dm-1.out‚ 7987.4765625% disk./.usageGB‚232.17800521850586 disk.sda.in‚ 0.06640625 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.06640625 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 disk.dm-0.in‚ 395.65234375# disk.dm-0.out‚33536.10791015625 disk.rbd1.in‚0 disk.nvme0n1.in‚ 395.65234375( proc.memory.availableMB‚ 780.12109375) proc.memory.percent‚16.38016700744629 network.recv‚ 2.360754e+06* disk./.usagePercent‚15.581660740535996 disk.drbd2265.out‚0 disk.sda1.out‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0& disk.nvme0n1.out‚33536.10791015625$ memory_percent‚95.23851871490479 proc.cpu.threads‚17 disk.rbd1.out‚ 15.2978515625 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.sda3.in‚ 0.06640625" disk.nvme0c0n1.in‚ 395.65234375 disk.sda2.out‚0( disk.nvme0c0n1.out‚33536.10791015625" proc.memory.rssMB‚ 2683.7265625 cpu‚25.34850370869758 disk.sda.out‚ 7987.47656251à„’!‚(:‡ ‚³ãÔÿÇ® gpu.0.gpu‚15.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚153.974' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.7948 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0OS̓!‚(:ø ‚³ãÔɾ‡±% disk.nvme0n1.out‚33539.4990234375 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0) proc.memory.percent‚16.38016700744629 disk.sda1.out‚0" proc.memory.rssMB‚ 2683.7265625% disk./.usageGB‚232.17800521850586 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.06640625" disk.nvme0c0n1.in‚ 395.65234375 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.06640625# memory_percent‚95.2355146408081( proc.memory.availableMB‚ 780.61328125 cpu‚25.358393583128443' disk.nvme0c0n1.out‚33539.4990234375 disk.sda.in‚ 0.06640625 disk.drbd2080.in‚0 disk.rbd0.in‚0 network.sent‚ 2.0804716e+07 disk.sda.out‚ 7988.09765625 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 7988.09765625 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 395.65234375 disk.sda3.out‚ 7988.09765625* disk./.usagePercent‚15.581660740535996 disk.rbd1.out‚ 15.2978515625 disk.drbd2266.out‚0" disk.dm-0.out‚33539.4990234375 proc.cpu.threads‚17 network.recv‚ 2.364273e+06 disk.nvme0n1.in‚ 395.65234375Š( ‘‘!‚(:† ‘³ãÔ¸£­® gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚122.154' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¯ÆH€|’!‚(:ñ ‘³ãÔˆ£ü± disk.rbd1.out‚ 15.2978515625 disk.drbd2080.out‚0 disk.dm-0.in‚ 395.65234375 cpu‚25.369425024558637 disk.drbd2266.in‚0 disk.sda2.out‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 780.35546875 disk.drbd2265.in‚0 disk.rbd0.out‚0 proc.cpu.threads‚17 network.recv‚ 2.368642e+06 disk.sda.out‚ 7988.62109375 disk.drbd1716.in‚0% disk./.usageGB‚232.17812728881836 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2266.out‚0 disk.rbd0.in‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 395.65234375 proc.memory.rssMB‚ 2684.15625 disk.sda3.in‚ 0.06640625 disk.sda3.out‚ 7988.62109375 network.sent‚ 2.0810683e+07 disk.sda.in‚ 0.06640625 disk.sda2.in‚0 disk.dm-1.out‚ 7988.62109375 disk.sda1.in‚0# disk.nvme0n1.out‚33539.92578125) disk./.usagePercent‚15.58166893277709 disk.drbd2265.out‚0 disk.dm-0.out‚33539.92578125 disk.nvme0n1.in‚ 395.65234375$ memory_percent‚95.23708820343018* proc.memory.percent‚16.382789611816406 disk.drbd2080.in‚0% disk.nvme0c0n1.out‚33539.92578125P‹x‡2“!- _wandbruntime‚6692  _runtime‚6692Þb¬7T*”!ÎT _runtime‚6692.053009422 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚250818445767680 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1006 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚48087225 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚239144958589380 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚199 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 410712064 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 127308288 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚265249930641920 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚416 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚416 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2460 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚48087503 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚253101129305088 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 111728565 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 171226212 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚26 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚11942590413312 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 111728567 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚985 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0^¦lÜY*•!ÓT OADeviceStatsMonitor.on_validation_batch_start/active.all.allocated‚ 282954779 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 111728565 PJDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚238875982662656 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚65222634 [MDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.current‚ 131512832 OIDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.current‚0 LFDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.peak‚0 aMDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.freed‚227236918631424 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚217 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 113309857 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 113310137 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚12148932849664 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.freed‚0 _step‚694 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚238876000665600 J@DeviceStatsMonitor.on_validation_batch_start/allocation.all.peak‚24401 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 5041553408 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ^JDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.freed‚238875982662656 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 282938800 \NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.current‚ 417333248 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚217 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚199 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚199 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚250818445767680 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 410712064 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 171210235 eRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.freed‚12148825276928 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚265250062154752 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ZKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚65222632 XJDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.peak‚ 269877760 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884020674157543e+09 BDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2460 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚48087503 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚253101129305088 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 VHDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.freed‚ 111728565 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_validation_batch_start/oversize_segments.peak‚0 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 171226212 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚26 H@DeviceStatsMonitor.on_validation_batch_start/segment.all.current‚416 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚11942590413312 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 111728567 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚985 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 _runtime‚6692.053009422 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚250818445767680 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1006 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚48087225 ZFDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.freed‚239144958589380 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 QIDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.allocated‚199 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 410712064 ]ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.current‚ 127308288 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚265249930641920 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚416 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚416 FDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 131512832 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 145311232 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚26 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚279 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 269877760 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚239163348714332 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 111739279 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚12149174412800 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚199 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚239163207749380 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 417333248 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 417333248 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248Û ì c)˜!ÝR QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚985 ZMDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.current‚17039360 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 145311232 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 107572736 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚416 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 YKDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.current‚ 131512832 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 145311232 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 PIDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.peak‚26 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚279 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 269877760 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚239163348714332 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 111739279 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚12149174412800 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚199 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚239163207749380 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 417333248 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 417333248 RLDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.current‚2 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚11942939549184 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 5041553408 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚250837660089344 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚416 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚11942812240896 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 5041553408 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 4541568000 cODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.allocated‚238894865851392 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚11908387564292 OGDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.allocated‚199 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 4912020992 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_validation_batch_end/active.small_pool.freed‚ 171217631 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚11942939549184 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚11942812240896 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚12149281985536 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 140964952 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 5041553408 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 171233608 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 113320829 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 111739277 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚227254820185088 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 127308288 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚65229132 trainer/global_step‚8462 % _timestamp‚1.7884020683332763e+09 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 282956908 TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 111739277 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2460 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 JBDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.peak‚217 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 282972887 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 113320548 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1006 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 4757245257 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚217 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚217 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 410712064 MEDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.current‚281 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚227254837224448 XJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.peak‚ 406789969 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 LFDeviceStatsMonitor.on_validation_batch_end/oversize_segments.allocated‚0 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚416 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚238894847848448 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 171217631 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚48091695 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.freed‚48091416 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚253120732870656 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚253120708930560 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚11908511489884 [MDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.current‚ 123925592 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 282972887 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚265269883343360 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2460 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚65229134 _step‚695 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 410712064 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚265270014856192 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚250837805400576 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚238894865851392 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚238894847848448 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 171233608 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 B&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 )loss/val_total_loss‚44.72901922633451 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24973811209201813 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13467814028263092 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.493885338306427 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.49205729365348816 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.040136564522981644 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5133609175682068 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.30537131428718567 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.001855287584476173 +jet/accuracy_2_of_3‚0.24132293462753296 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5055264830589294 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4870048761367798 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14551757276058197 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5203445553779602 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_2‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 jet/accuracy_5_of_6‚0 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.36283305287361145 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8232094645500183 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5977463722229004 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.10222223401069641 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.48999038338661194 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5133667588233948 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3642857074737549 particle/accuracy_1_of_5‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15225259959697723 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13078762590885162 particle/accuracy_2_of_3‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5310814380645752 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5076023936271667 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8610478639602661 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.28884950280189514 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11949841678142548 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9929143190383911 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5583983659744263 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2387387603521347 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 )jet/accuracy_1_of_4‚0.911659836769104 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5178571343421936 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2701011896133423 + Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4375 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09324963390827179 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5196492671966553 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4927152395248413 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9444443583488464 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_5‚1 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1818181723356247 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4548611640930176 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9776114225387573 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9872446656227112 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_6‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5413000583648682 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14225448668003082 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.017241379246115685 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.386799693107605 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.02160220593214035 ,particle/sensitivity‚0.27935919165611267 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.971875011920929 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3693781793117523 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9856817126274109 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.514162540435791 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.469527006149292 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9908238053321838 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1083853617310524 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0033468513283878565 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 particle/accuracy_4_of_6‚1 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2222222238779068 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4281046390533447 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4005590081214905 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.08749999105930328 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5669312477111816 jet/accuracy_6_of_6‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3940545320510864 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5709876418113708 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.26804041862487793 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2583128809928894 <²î PHf"Purity/3FRt*SRqqt*FBt/event_purity‚0.0054644811898469925 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4076307713985443 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9823307394981384 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21415896713733673 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1457640528678894 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13951389491558075 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9935541152954102 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5021623969078064 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1511545032262802 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9903486371040344 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5844780206680298 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.20075757801532745 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 particle/accuracy_3_of_6‚1 jet/accuracy_3_of_7‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.03333333507180214 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28425928950309753 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975137710571289 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07156078517436981 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.48084205389022827 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5543993711471558 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0008818342466838658 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9479167461395264 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21004854142665863 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9789473414421082 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.09029697626829147 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.337037056684494 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.17978395521640778 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25172415375709534 particle/accuracy_1_of_1‚1 jet/accuracy_7_of_7‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02706291526556015 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19005891680717468 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1891026794910431 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12493888288736343 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_3_of_4‚0.06250939518213272 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.35555556416511536 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9462360739707947 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3518390357494354 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2867063283920288 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9807692170143127 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16707420349121094 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5421841740608215 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.05094339698553085 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5775862336158752 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4109981060028076 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4139542579650879 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5133667588233948 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.35451388359069824 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 jet/accuracy_6_of_7‚0 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.115079365670681 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1490897685289383 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.39827588200569153 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10670509934425354 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16604653000831604 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20407797396183014 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 +jet/accuracy_3_of_6‚0.17307692766189575 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9255319237709045 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5220175385475159 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03209247067570686 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5535714030265808 particle/accuracy_7_of_7‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7validation_average_jet_accuracy‚0.43479636311531067 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.971230149269104 +jet/accuracy_2_of_2‚0.10839098691940308 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 _step‚696 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 jet/accuracy_5_of_5‚0 particle/accuracy_3_of_3‚1 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_1_of_6‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.996494710445404 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 _runtime‚ 6705.2887403 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1866985410451889 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5687087178230286 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.49791669845581055 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2222222238779068 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5416666269302368 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014992942102253437 particle/accuracy_1_of_3‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3351851999759674 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 % _timestamp‚1.7884020806496747e+09 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.008602150715887547 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9945802688598633 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5407229661941528 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 epoch‚6 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.09674329310655594 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 trainer/global_step‚8462 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.11642710119485855 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5416666269302368 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16604653000831604 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 *jet/accuracy_1_of_2‚0.7509299516677856 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3055555522441864 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16581261157989502 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 jet/accuracy_1_of_7‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.39263391494750977 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5196428894996643 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09772045165300369 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2774113416671753 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5485151410102844 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16751882433891296 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.406988263130188 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 jet/accuracy_4_of_6‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4912588894367218 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3181818127632141 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5133333206176758 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_4_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07581211626529694 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_7‚1 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3953723907470703 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1660919487476349 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.36245790123939514 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.183866947889328 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6397684812545776 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5460467338562012 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,jet/accuracy_4_of_5‚0.001792114693671465 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4462150037288666 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9869614839553833 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4912356436252594 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.0347786583006382 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1813810020685196 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0007936508045531809 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_7‚1 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 (particle/accuracy‚0.8420862555503845 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4714215099811554 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5067523717880249 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20187342166900635 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 'particle/f_score‚0.3556472361087799 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.38271602988243103 particle/accuracy_5_of_7‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9974799752235413 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.41366884112358093 particle/accuracy_1_of_4‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9747664332389832 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5532450675964355 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.990651547908783 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14787618815898895 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3958333432674408 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5220230221748352 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.46024397015571594 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3972868025302887 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.031836699694395065 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4364311397075653 particle/accuracy_4_of_4‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.383507639169693 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.47482675313949585 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4764558672904968 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.552880585193634 +jet/accuracy_2_of_6‚0.46794870495796204 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13341592252254486 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4829048812389374 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.17426900565624237 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05921052768826485 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5490735173225403 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.38174375891685486 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09219858795404434 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3243093192577362 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07537563145160675 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01973682828247547 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.158987358212471 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5065199732780457 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3055555522441864 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9481438994407654 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740740299224854 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9853872060775757 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5039536356925964 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1775956004858017 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.42777782678604126 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9901475310325623 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5313761234283447 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.07098766416311264 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9887130260467529 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010571014136075974 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24024900794029236 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2902108132839203 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08408594876527786 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973471164703369 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26097574830055237 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004913076292723417 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.16543583571910858 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5725694298744202 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2916666865348816 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_7‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9886366128921509 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.022656839340925217 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.977729856967926 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5810185670852661 particle/accuracy_6_of_6‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4566666781902313 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6891366839408875 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.534923255443573 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5362812280654907 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7424241900444031•Ùûbª!ï¦ 8"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.493885338306427 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.49205729365348816 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.040136564522981644 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/3FRt0SRqqt*FBt/event_purity‚0.01666666753590107 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5133609175682068 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.30537131428718567 ;"Purity/2FRt*SRqqt0FBt/event_purity‚0.001855287584476173 +jet/accuracy_2_of_3‚0.24132293462753296 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_7‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5055264830589294 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4870048761367798 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.14551757276058197 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5203445553779602 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_2‚1 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2708333134651184 jet/accuracy_5_of_6‚0 *"Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0.5 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.36283305287361145 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8232094645500183 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.5977463722229004 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.10222223401069641 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.48999038338661194 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5133667588233948 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3642857074737549 particle/accuracy_1_of_5‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 *"Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0.5 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.15225259959697723 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13078762590885162 particle/accuracy_2_of_3‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5310814380645752 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5076023936271667 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8610478639602661 -"Purity/3FRt0SRqqt1FBt/event_purity‚0.0625 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.28884950280189514 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.11949841678142548 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9929143190383911 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5583983659744263 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2387387603521347 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 )jet/accuracy_1_of_4‚0.911659836769104 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5178571343421936 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2701011896133423 + Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4375 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09324963390827179 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5196492671966553 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4927152395248413 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9444443583488464 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_5‚1 7 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.1818181723356247 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4548611640930176 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 +"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.75 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9776114225387573 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9872446656227112 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 ,&Pur.&nÌyity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_5_of_6‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5413000583648682 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ( Purity/2FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14225448668003082 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.017241379246115685 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 6 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.386799693107605 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 +jet/accuracy_3_of_3‚0.02160220593214035 ,particle/sensitivity‚0.27935919165611267 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.971875011920929 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_4‚0.3693781793117523 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9856817126274109 8"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.514162540435791 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.469527006149292 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9908238053321838 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt0SRqqt2FBt/event_purity‚0.1083853617310524 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 -jet/accuracy_4_of_4‚0.0033468513283878565 particle/accuracy_2_of_5‚1 particle/accuracy_3_of_5‚1 particle/accuracy_4_of_6‚1 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2222222238779068 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4281046390533447 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 9"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.4005590081214905 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.08749999105930328 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5669312477111816 jet/accuracy_6_of_6‚0 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.3940545320510864 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5709876418113708 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 particle/accuracy_2_of_6‚1 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.26804041862487793 particle/accuracy_1_of_2‚1 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2583128809928894 <"Purity/3FRt*SRqqt*FBt/event_purity‚0.0054644811898469925 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_6_of_7‚1 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4076307713985443 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9823307394981384 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.21415896713733673 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.1457640528678894 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 :"Purity/*FRt0SRqqt0FBt/event_purity‚0.13951389491558075 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.6875 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.7142857313156128 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9935541152954102 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5021623969078064 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_2_of_7‚0 7 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.1511545032262802 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9903486371040344 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5844780206680298 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.20075757801532745 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 particle/accuracy_3_of_6‚1 jet/accuracy_3_of_7‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 :"Purity/3FRt*SRqqt1FBt/event_purity‚0.03333333507180214 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.28425928950309753 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 5 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.80952388048172 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9975137710571289 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.07156078517436981 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 8"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.800000011920929 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.48084205389022827 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5543993711471558 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0008818342466838658 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9479167461395264 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21004854142665863 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9789473414421082 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 +jet/accuracy_3_of_5‚0.09029697626829147 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 6 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.337037056684494 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.17978395521640778 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.25172415375709534 particle/accuracy_1_of_1‚1 jet/accuracy_7_of_7‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 :"Purity/2FRt0SRqqt*FBt/event_purity‚0.02706291526556015 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.19005891680717468 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.1891026794910431 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.12493888288736343 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 +jet/accuracy_3_of_4‚0.06250939518213272 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.35555556416511536 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 +particle/specificity‚0.9462360739707947 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.3518390357494354 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2867063283920288 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_6‚0.9807692170143127 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16707420349121094 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5421841740608215 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.05094339698553085 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5775862336158752 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4109981060028076 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4139542579650879 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5133667588233948 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.35451388359069824 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 jet/accuracy_6_of_7‚0 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.115079365670681 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1490897685289383 :"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.39827588200569153 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10670509934425354 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.16604653000831604 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20407797396183014 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 +jet/accuracy_3_of_6‚0.17307692766189575 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9255319237709045 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5220175385475159 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03209247067570686 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5535714030265808 particle/accuracy_7_of_7‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7validation_average_jet_accuracy‚0.43479636311531067 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.971230149269104 +jet/accuracy_2_of_2‚0.10839098691940308 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 _step‚696 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 jet/accuracy_5_of_5‚0 particle/accuracy_3_of_3‚1 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 particle/accuracy_1_of_6‚1 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 6 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.996494710445404 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 _runtime‚ 6705.2887403 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1866985410451889 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5687087178230286 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 :"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.49791669845581055 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2222222238779068 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5416666269302368 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.014992942102253437 particle/accuracy_1_of_3‚1 7 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.3351851999759674 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 % _timestamp‚1.7884020806496747e+09 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.008602150715887547 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9945802688598633 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5407229661941528 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 epoch‚6 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.09674329310655594 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 trainer/global_step‚8462 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.11642710119485855 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5416666269302368 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.16604653000831604 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 *jet/accuracy_1_of_2‚0.7509299516677856 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3055555522441864 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16581261157989502 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 jet/accuracy_1_of_7‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.39263391494750977 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5196428894996643 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.09772045165300369 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2774113416671753 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5485151410102844 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.16751882433891296 6 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.406988263130188 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 jet/accuracy_4_of_6‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4912588894367218 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.3181818127632141 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.5133333206176758 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_4_of_7‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ( Purity/2FRt0SRqqt3FBt/FRt_purity‚0.5 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07581211626529694 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_3_of_7‚1 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.3953723907470703 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.1660919487476349 :"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.36245790123939514 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 6 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.183866947889328 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6397684812545776 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5460467338562012 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,jet/accuracy_4_of_5‚0.001792114693671465 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 7 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.4462150037288666 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9869614839553833 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.4912356436252594 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt2SRqqt0FBt/event_purity‚0.0347786583006382 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1813810020685196 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0007936508045531809 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_7‚1 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 (particle/accuracy‚0.8420862555503845 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4714215099811554 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5067523717880249 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 jet/accuracy_5_of_7‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_4‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.20187342166900635 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 7 Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1666666716337204 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 'particle/f_score‚0.3556472361087799 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.38271602988243103 particle/accuracy_5_of_7‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9974799752235413 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.41366884112358093 particle/accuracy_1_of_4‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9747664332389832 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8518518209457397 +"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.75 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5532450675964355 6 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.990651547908783 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.14787618815898895 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3958333432674408 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5220230221748352 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.46024397015571594 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3972868025302887 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.8333333134651184 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.031836699694395065 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 *jet/accuracy_2_of_5‚0.4364311397075653 particle/accuracy_4_of_4‚1 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3571428656578064 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.383507639169693 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.47482675313949585 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4764558672904968 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 8"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.552880585193634 +jet/accuracy_2_of_6‚0.46794870495796204 8 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.13341592252254486 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4829048812389374 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.17426900565624237 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05921052768826485 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5490735173225403 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.38174375891685486 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 ,"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.375 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09219858795404434 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3243093192577362 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07537563145160675 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01973682828247547 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 8"Purity/0FRt1SRqqt1FBt/event_purity‚0.158987358212471 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5065199732780457 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3055555522441864 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.2380952537059784 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9481438994407654 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740740299224854 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9853872060775757 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5039536356925964 7 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.1775956004858017 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.42777782678604126 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9901475310325623 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5313761234283447 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.07098766416311264 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9887130260467529 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.010571014136075974 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.24024900794029236 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2902108132839203 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08408594876527786 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973471164703369 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26097574830055237 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.004913076292723417 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.16543583571910858 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5725694298744202 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.2916666865348816 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_7‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9886366128921509 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.022656839340925217 6 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.977729856967926 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5810185670852661 particle/accuracy_6_of_6‚1 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4566666781902313 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6891366839408875 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1666666716337204 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.534923255443573 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5362812280654907 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7424241900444031 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03488372266292572 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9898778200149536 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735631346702576 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 9"Purity/*FRt1SRqqt*FBt/event_purity‚0.2207571566104889 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 )loss/val_total_loss‚44.72901922633451 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.24973811209201813 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.13467814028263092 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208¸3êC’ž!‚(:‡ ¯³ãÔÁÊ°® gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚122.684' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.5368 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¥üؤrŸ!‚(:ç ¯³ãÔŒŒ»² disk.drbd2080.out‚0" disk.dm-0.out‚33663.8916015625 disk.sda1.in‚0 disk.rbd1.out‚ 15.2978515625" disk.nvme0c0n1.in‚ 395.65234375 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.dm-1.in‚ 0.06640625 disk.dm-1.out‚7989.625 disk.rbd0.out‚0% disk.nvme0n1.out‚33663.8916015625 disk.sda.out‚7989.625 disk.sda2.in‚0$ disk./.usageGB‚232.1785888671875 disk.drbd2265.in‚0 disk.sda2.out‚0* disk./.usagePercent‚15.581699909688734 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 395.65234375 disk.sda3.out‚7989.625' disk.nvme0c0n1.out‚33663.8916015625 disk.sda.in‚ 0.06640625 network.recv‚ 2.379441e+06 disk.sda3.in‚ 0.06640625 disk.drbd2266.in‚0 disk.drbd2266.out‚0( proc.memory.availableMB‚ 779.34765625 network.sent‚ 2.0925419e+07 disk.drbd1716.out‚0" memory_percent‚95.243239402771! proc.memory.rssMB‚ 2685.171875 cpu‚25.379193997928326 disk.rbd1.in‚0 disk.nvme0n1.in‚ 395.65234375 disk.drbd1716.in‚0* proc.memory.percent‚16.388988494873047¯ó¿g !- _wandbrunwžxVtime‚6733  _runtime‚6733KjÈ¡! trainer/global_step‚8499 $ _timestamp‚1.788402109136416e+09 _runtime‚6733.773347213 _step‚697 ' lr-AdamW/pg1‚5.2781974622007084e-05 ' lr-AdamW/pg2‚5.2781974622007084e-05Q4{¬Í¢!Ç ' lr-AdamW/pg1‚5.2781974622007084e-05 ' lr-AdamW/pg2‚5.2781974622007084e-05 trainer/global_step‚8499 $ _timestamp‚1.788402109136416e+09 _runtime‚6733.773347213 _step‚697¹0 šÊ2£!- _wandbruntime‚6733  _runtime‚6733ø§¹‘ì'¤!æO QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 trainer/global_step‚8499 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252119270327808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚240121810397184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48337175 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚228421511986176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚228421494946816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 _step‚698 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172047842 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚252119270327808 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚65579204 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚240121792394240 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚266621415260160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚254416445359104 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚240384383722174 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11997477933568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112334259 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172047842 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚252119457330688 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 172027804 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284382101 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788402109137105e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112334259 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11962888775358 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65579206 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48336662 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 284362061 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚515 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12204969901056 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚240384565882938 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚240121792394240 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚240384383722174 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚11997477933568 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 112334259 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 172047842 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚252119457330688 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 172027804 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 284382101 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 $ _timestamp‚1.788402109137105e+09 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112334259 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚11962888775358 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚65579206 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48336662 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 284362061 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚515 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12204969901056 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚240384565882938 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚240121792394240 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚252119270327808 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚240121810397184 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48337175 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚228421511986176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚228421494946816 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 _step‚698 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 172047842 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760ºœ¦j2¦!- _wandbruntime‚6734  _runtime‚6734U³­‚÷&§!ñM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚240149079761920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112346918 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 % _timestamp‚1.7884021098182118e+09 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚252148119846912 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172047591 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113928521 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284414573 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113929059 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65586581 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172067655 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172067655 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 284394507 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚266651505474048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11998871071744 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚266651742082560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚228447438129152 _step‚699 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11964277939902 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48342476 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48341940 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11964443061514 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12206383775232 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚252148119846912 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65586583 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112346916 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172047591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8499-ï!²Ö¨!öM UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112346918 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 % _timestamp‚1.7884021098182118e+09 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C=DeviceStatsMonitor.on_train_batch_end/Ç ¢‡& oversize_segments.freed‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚252148119846912 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 172047591 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 113928521 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 284414573 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 113929059 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65586581 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 172067655 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 172067655 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 284394507 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚266651505474048 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚11998871071744 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚266651742082560 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚228447438129152 _step‚699 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚11964277939902 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48342476 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48341940 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚536 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚11964443061514 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12206383775232 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚252148119846912 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65586583 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112346916 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 172047591 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 trainer/global_step‚8499 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚240149079761920 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚538 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760»y œÑ2©!- _wandbruntime‚6734  _runtime‚6734ºn¨EUª!Ï /loss/FRt2/assignment_loss‚3.904378664493561 0loss/FBt4/assignment_loss‚0.1641657109035929 /loss/FBt4/detection_loss‚0.6318137212338942 2loss/SRqqt1/assignment_loss‚1.7468652535950946 /loss/FBt2/detection_loss‚2.4771633382354463 trainer/global_step‚8499 1loss/SRqqt1/detection_loss‚1.6493527684961595 /loss/SRqqt3/detection_loss‚1.69484909872214 /loss/FBt1/detection_loss‚2.2683134216528673 # _timestamp‚1.78840210981999e+09 0loss/FBt1/assignment_loss‚0.2551454481532136 .loss/FBt3/detection_loss‚2.773490476608276 epoch‚7 0loss/FRt1/assignment_loss‚2.6627178010293995 /loss/FRt3/assignment_loss‚2.504216805100441 2loss/SRqqt2/assignment_loss‚1.9261314616753504 2loss/SRqqt3/assignment_loss‚1.7578339715506517 2loss/SRqqt4/assignment_loss‚0.9877443584045977 1loss/FBt3/assignment_loss‚0.14064877837590686 0loss/FRt4/assignment_loss‚1.3202112650871276 0loss/FBt2/assignment_loss‚0.4337344105751494 .loss/FRt3/detection_loss‚2.723545227665454 1loss/SRqqt4/detection_loss‚1.0493034175015055 /loss/FRt1/detection_loss‚1.9773214393753116 -loss/FRt2/detection_loss‚3.00721950729688 0loss/SRqqt2/detection_loss‚2.327710052331289 %loss/total_loss‚42.35391624689758 _runtime‚6734.456742105 /loss/FRt4/detection_loss‚1.9700398488342763 _step‚700,ÌZ«!Ô 0loss/FBt1/assignment_loss‚0.2551454481532136 .loss/FBt3/detection_loss‚2.773490476608276 epoch‚7 0loss/FRt1/assignment_loss‚2.6627178010293995 /loss/FRt3/assignment_loss‚2.504216805100441 2loss/SRqqt2/assignment_loss‚1.9261314616753504 2loss/SRqqt3/assignment_loss‚1.7578339715506517 2loss/SRqqt4/assignment_loss‚0.9877443584045977 1loss/FBt3/assignment_loss‚0.14064877837590686 0loss/FRt4/assignment_loss‚1.3202112650871276 0loss/FBt2/assignment_loss‚0.4337344105751494 .loss/FRt3/detection_loss‚2.723545227665454 1loss/SRqqt4/detection_loss‚1.0493034175015055 /loss/FRt1/detection_loss‚1.9773214393753116 -loss/FRt2/detection_loss‚3.00721950729688 0loss/SRqqt2/detection_loss‚2.327710052331289 %loss/total_loss‚42.35391624689758 _runtime‚6734.456742105 /loss/FRt4/detection_loss‚1.9700398488342763 _step‚700 /loss/FRt2/assignment_loss‚3.904378664493561 0loss/FBt4/assignment_loss‚0.1641657109035929 /loss/FBt4/detection_loss‚0.6318137212338942 2loss/SRqqt1/assignment_loss‚1.7468652535950946 /loss/FBt2/detection_loss‚2.4771633382354463 trainer/global_step‚8499 1loss/SRqqt1/detection_loss‚1.6493527684961595 /loss/SRqqt3/detection_loss‚1.69484909872214 /loss/FBt1/detection_loss‚2.2683134216528673 # _timestamp‚1.78840210981999e+09¼3¯´¿¬!‚(:’ ¾³ãÔ’Ç’® gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚135.089' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.017799999999998 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÜÄÃ΄­!‚(:ù ¾³ãÔ¯ó€±( disk.nvme0c0n1.out‚33787.55224609375% disk./.usageGB‚232.17811965942383 disk.drbd2265.out‚0 disk.drbd2080.out‚0# disk.dm-0.out‚33787.55224609375 disk.drbd2266.in‚0$ memory_percent‚95.24481296539307 network.sent‚ 2.1049021e+07* disk./.usagePercent‚15.581668420762023 disk.drbd2080.in‚0 disk.rbd1.out‚ 15.2978515625& disk.nvme0n1.out‚33787.55224609375! proc.memory.rssMB‚ 2685.171875 disk.sda1.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 395.65234375 disk.sda3.out‚ 7990.3359375 disk.sda.out‚ 7990.3359375 disk.dm-1.out‚ 7990.3359375 disk.sda2.in‚0 disk.rbd0.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.06640625" disk.nvme0c0n1.in‚ 395.65234375 cpu‚25.378437866277192 proc.cpu.threads‚17 network.recv‚ 2.384751e+06 disk.nvme0n1.in‚ 395.65234375 disk.sda.in‚ 0.06640625 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.sda2.out‚0* proc.memory.percent‚16.388988494873047 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.06640625( proc.memory.availableMB‚ 779.08984375Û9§®!‚(:’ ͳãÔ§Þ˜® gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚124.287' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.857400000000002 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²ñ„¯!‚(:ù ͳãÔ˜·Ý± disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda.out‚ 7991.0859375 disk.drbd1716.in‚0 network.sent‚ 2.1157317e+07* disk./.usagePercent‚15.581704261816817 disk.sda.in‚ 0.06640625$ memory_percent‚95.24357318878174 disk.sda2.out‚0 network.recv‚ 2.390658e+06 disk.dm-1.out‚ 7991.0859375 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 7991.0859375( disk.nvme0c0n1.out‚33793.05517578125 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 395.65234375# disk.dm-0.out‚33793.05517578125 disk.sda2.in‚0! proc.memory.rssMB‚ 2685.171875 cpu‚25.377705199557738 proc.cpu.threads‚17% disk./.usageGB‚232.17865371704102 disk.rbd0.out‚0 disk.sda3.in‚ 0.06640625 disk.rbd1.out‚ 15.2978515625 disk.nvme0n1.in‚ 395.65234375& disk.nvme0n1.out‚33793.05517578125* proc.memory.percent‚16.388988494873047 disk.dm-1.in‚ 0.06640625 disk.drbd2266.in‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 395.65234375 disk.drbd1716.out‚0( proc.memory.availableMB‚ 779.29296875Î>Nð!‚(:… ܳãÔ‰›Ÿ® gpu.0.gpu‚24.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚132.13' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.426 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û þ(±!‚(:ö ܳãÔ²òŸ±% disk.nvme0n1.out‚33802.4677734375 disk.dm-1.in‚ 0.06640625 disk.drbd2265.out‚0# memory_percent‚95.9984302520752" disk.nvme0c0n1.in‚ 395.66015625 disk.drbd1716.in‚0! proc.memory.rssMB‚ 2685.171875 proc.cpu.threads‚17 disk.rbd0.out‚0 network.recv‚ 2.394614e+06* disk./.usagePercent‚15.581784136167498 disk.sda.in‚ 0.06640625 disk.sda2.in‚0 disk.dm-1.out‚ 7993.49609375$ disk./.usageGB‚232.1798439025879 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 15.2978515625 cpu‚25.376911349946955 disk.sda.out‚ 7993.49609375 disk.dm-0.in‚ 395.66015625 disk.sda3.in‚ 0.06640625' disk.nvme0c0n1.out‚33802.4677734375 disk.nvme0n1.in‚ 395.66015625 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 7993.49609375' proc.memory.availableMB‚ 655.6171875 disk.rbd0.in‚0" disk.dm-0.out‚33802.4677734375 network.sent‚ 2.1163273e+07 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.drbd2266.out‚0* proc.memory.percent‚16.388988494873047Åéš2²!- _wandbruntime‚6769  _runtime‚6769‚ÖÄƳ!À & lr-AdamW/pg1‚5.145197086335186e-05 & lr-AdamW/pg2‚5.145197086335186e-05 trainer/global_step‚8549 $ _timestamp‚1.788402145062944e+09 _runtime‚6769.699867242 _step‚701#îòË´!Å trainer/global_step‚8549 $ _timestamp‚1.788402145062944e+09 _runtime‚6769.699867242 _step‚701 & lr-AdamW/pg1‚5.145197086335186e-05 & lr-AdamW/pg2‚5.145197086335186e-05½åƒ€2µ!- _wandbruntime‚6769  _runtime‚6769 *Ó‚í'¶!çO OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48606281 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114554335 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚229718654063616 trainer/global_step‚8549 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚268137933041152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12275629686272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚241751014657382 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚241485278633984 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚253552596890112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286007753 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 114554876 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚255862068830208 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚241751196818146 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12067149256192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚229718671102976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚253552409887232 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112967209 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚702 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚241485260631040 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112967207 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚268137698516480 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48606820 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12032360593766 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚253552409887232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217.̽T·!ìO UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚268137933041152 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12275629686272 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚241751014657382 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚241485278633984 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚253552596890112 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 286007753 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 114554876 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚255862068830208 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚241751196818146 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12067149256192 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚229718671102976 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚253552409887232 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 112967209 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚702 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚241485260631040 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 112967207 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚268137698516480 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48606820 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12032360593766 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚253552409887232 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚48606281 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 114554335 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚229718654063616 trainer/global_step‚8549¾ènù2¸!- _wandbruntime‚6770  _runtime‚6770™ê™€÷&¹!ñM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚255891005718528 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173060357 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286040225 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12277048727552 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 173060357 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112979866 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114567574 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚517 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12277259298816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚241512529995776 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚268168265017344 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 286040225 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12068542394368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65955433 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚253581072390144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65955431 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚253581259406336 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114567057 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112979868 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48611626 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚703 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12033749758310 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚241778529165234 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12033914879922 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12068542394368 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229744614285312 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112979866 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚241512547998720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚253581259406336 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112979868 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 173040293 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚241778347004262 _runtime‚6770.371435035 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 286020159 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884021457340777e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173040293 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚241512529995776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12068711407616 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 286020159 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚241512547998720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚515 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚255890981778432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972óEÊü&º!öM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 112979866 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 114567574 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚517 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12277259298816 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚241512529995776 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚268168265017344 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 286040225 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12068542394368 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚65955433 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚253581072390144 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚65955431 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚253581259406336 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 114567057 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 112979868 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚48611626 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _step‚703 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12033749758310 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚241778529165234 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12033914879922 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12068542394368 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚229744614285312 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 112979866 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚241512547998720 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚253581259406336 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 112979868 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 173040293 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚241778347004262 _runtime‚6770.371435035 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 286020159 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884021457340777e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 173040293 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚241512529995776 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12068711407616 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 286020159 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚241512547998720 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚515 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚255890981778432 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚255891005718528 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 173060357 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 286040225 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12277048727552 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 173060357 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217¿œ`÷³2»!- _wandbruntime‚6770  _runtime‚6770Dz;_[¼!Õ 2loss/SRqqt1/assignment_loss‚1.4914104667993693 1loss/SRqqt2/assignment_loss‚2.072309732437134 /loss/FRt1/detection_loss‚2.0371024385094643 .loss/FRt3/detection_loss‚2.565075728722981 /loss/FBt4/detection_loss‚0.9008540131400029 /loss/FRt2/assignment_loss‚4.142926620585578 0loss/FRt4/assignment_loss‚1.5312388064497608 0loss/FBt2/assignment_loss‚0.2315190215420131 1loss/SRqqt2/detection_loss‚2.5024397019986755 epoch‚7 /loss/FRt1/assignment_loss‚2.772814425475457 /loss/FRt2/detection_loss‚3.0031410540853227 .loss/FRt4/detection_loss‚1.742187518689592 2loss/FBt3/assignment_loss‚0.018009432152509497 1loss/SRqqt1/detection_loss‚1.4419014050410344 1loss/SRqqt4/detection_loss‚0.9698563426324766 .loss/FBt1/detection_loss‚2.132488783862856 &loss/total_loss‚42.516879619886055 2loss/SRqqt4/assignment_loss‚0.9469861571919428 1loss/FBt4/assignment_loss‚0.12274704245196517 trainer/global_step‚8549 1loss/SRqqt3/assignment_loss‚1.149072031180064 1loss/SRqqt3/detection_loss‚1.6085205329091925 /loss/FBt2/detection_loss‚2.5625970400869846 _runtime‚6770.372467563 0loss/FBt1/assignment_loss‚0.3174783279601898 /loss/FBt3/detection_loss‚2.6866525411605835 $ _timestamp‚1.788402145735723e+09 _step‚704 0loss/FRt3/assignment_loss‚3.5675504548209056 lj‰`½!Ú 1loss/SRqqt3/assignment_loss‚1.149072031180064 1loss/SRqqt3/detection_loss‚1.6085205329091925 /loss/FBt2/detection_loss‚2.5625970400869846 _runtime‚6770.372467563 0loss/FBt1/assignment_loss‚0.3174783279601898 /loss/FBt3/detection_loss‚2.6866525411605835 $ _timestamp‚1.788402145735723e+09 _step‚704 0loss/FRt3/assignment_loss‚3.5675504548209056 2loss/SRqqt1/assignment_loss‚1.4914104667993693 1loss/SRqqt2/assignment_loss‚2.072309732437134 /loss/FRt1/detection_loss‚2.0371024385094643 .loss/FRt3/detection_loss‚2.565075728722981 /loss/FBt4/detection_loss‚0.9008540131400029 /loss/FRt2/assignment_loss‚4.142926620585578 0loss/FRt4/assignment_loss‚1.5312388064497608 0loss/FBt2/assignment_loss‚0.2315190215420131 1loss/SRqqt2/detection_loss‚2.5024397019986755 epoch‚7 /loss/FRt1/assignment_loss‚2.772814425475457 /loss/FRt2/detection_loss‚3.0031410540853227 .loss/FRt4/detection_loss‚1.742187518689592 2loss/FBt3/assignment_loss‚0.018009432152509497 1loss/SRqqt1/detection_loss‚1.4419014050410344 1loss/SRqqt4/detection_loss‚0.9698563426324766 .loss/FBt1/detection_loss‚2.132488783862856 &loss/total_loss‚42.516879619886055 2loss/SRqqt4/assignment_loss‚0.9469861571919428 1loss/FBt4/assignment_loss‚0.12274704245196517 trainer/global_step‚8549ÀÛááG’¾!‚(:‡ ë³ãÔ´ë›° gpu.0.gpu‚42.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚140.751' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.1502 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀÔ£Ô…¿!‚(:ú ë³ãÔŸ¬«² network.sent‚ 2.1169598e+07 disk.drbd2265.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 15.8759765625 disk.sda.out‚ 7994.05859375 network.recv‚ 2.399289e+06 disk.dm-1.in‚ 0.06640625 disk.rbd0.in‚0 disk.sda3.out‚ 7994.05859375 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 7994.05859375 disk.drbd2080.out‚0 disk.sda2.out‚0( proc.memory.availableMB‚ 655.61328125* proc.memory.percent‚16.388988494873047 proc.cpu.threads‚17 disk.dm-0.in‚ 395.66015625# disk.dm-0.out‚33802.76220703125 disk.sda1.in‚0 disk.nvme0n1.in‚ 395.66015625& disk.nvme0n1.out‚33802.76220703125! proc.memory.rssMB‚ 2685.171875 cpu‚25.37618218958534 disk.sda.in‚ 0.06640625 disk.sda2.in‚0* disk./.usagePercent‚15.581787976280511% disk./.usageGB‚232.17990112304688 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0" disk.nvme0c0n1.in‚ 395.66015625 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.sda3.in‚ 0.06640625( disk.nvme0c0n1.out‚33802.76220703125# memory_percent‚95.9984540939331ôàPžœÀ!‚(:‘ ú³ãÔ”³…® gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚152.378' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.475599999999996 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æå# Á!‚(:ö ú³ãÔÈ* disk./.usagePercent‚15.581782856129827 disk.sda2.out‚0 disk.drbd2080.in‚0( proc.memory.availableMB‚ 655.87109375* proc.memory.percent‚16.388988494873047 disk.dm-1.in‚ 0.06640625" disk.nvme0c0n1.in‚ 395.66015625$ disk.nvme0n1.out‚33803.236328125 proc.cpu.threads‚17 disk.sda.in‚ 0.06640625 disk.sda2.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 15.8759765625& disk.nvme0c0n1.out‚33803.236328125 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0$ memory_percent‚95.99688053131104 network.recv‚ 2.404793e+06 disk.rbd1.in‚0! proc.memory.rssMB‚ 2685.171875 cpu‚25.375474553160462 network.sent‚ 2.1277566e+07% disk./.usageGB‚232.17982482910156 disk.sda.out‚ 7994.57421875 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0 disk.dm-0.in‚ 395.66015625 disk.dm-1.out‚ 7994.57421875 disk.drbd2266.in‚0 disk.drbd2080.out‚0! disk.dm-0.out‚33803.236328125 disk.sda1.out‚0 disk.sda3.out‚ 7994.57421875 disk.nvme0n1.in‚ 395.66015625 disk.sda3.in‚ 0.06640625PJ2Â!- _wandbruntime‚6805  _runtime‚6805Ö¹ÇÃ!Á _step‚705 & lr-AdamW/pg1‚5.013364603511327e-05 & lr-AdamW/pg2‚5.013364603511327e-05 trainer/global_step‚8599 % _timestamp‚1.7884021808437798e+09 _runtime‚6805.480630892`WÉ#ÌÄ!Æ _step‚705 & lr-AdamW/pg1‚5.013364603511327e-05 & lr-AdamW/pg2‚5.013364603511327e-05 trainer/global_step‚8599 % _timestamp‚1.7884021808437798e+09 _runtime‚6805.480630892Áé‘P2Å!- _wandbruntime‚6805  _runtime‚6805þA͵Æ!çO QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174013208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚243117645592590 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48876405 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12136989578752 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269654404482560 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 trainer/global_step‚8599 % _timestamp‚1.7884021808444865e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174013208 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚254985736449536 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12136820578816 VGDeviceStatsMonitor.on_train¶ø˜8_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚254985549446656 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12136820578816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚242848728867840 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚269654167860736 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243117827753354 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚242848746870784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174033246 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 115193311 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66316904 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113600157 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 287613365 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 115192779 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12101832412174 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚257307740181504 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12346451619328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12346664301056 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66316906 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚231015830219776 _runtime‚6805.481514018 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚254985549446656 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 287633405 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113600159 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚242848728867840 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚257307716241408 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113600159¢f™3ò'Ç!ìO HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 113600157 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 287613365 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 115192779 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12101832412174 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚257307740181504 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12346451619328 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12346664301056 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66316906 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚231015830219776 _runtime‚6805.481514018 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚254985549446656 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 287633405 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 113600159 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚242848728867840 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚257307716241408 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 113600159 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 174013208 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚243117645592590 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚48876405 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12136989578752 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚269654404482560 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 trainer/global_step‚8599 % _timestamp‚1.7884021808444865e+09 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 174013208 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚254985736449536 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12136820578816 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚254985549446656 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12136820578816 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚242848728867840 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚269654167860736 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚243117827753354 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚242848746870784 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 174033246 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 115193311 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66316904Ây“2È!- _wandbruntime‚6806  _runtime‚6806‘¦²÷&É!ñM ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚255014211949568 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚242875998232576 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚502 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚257336653129728 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚231041773402112 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚231041756362752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 113612818 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 trainer/global_step‚8599 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884021815125117e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6806.149939617 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113612816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287665877 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚243144977939470 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12138382730240 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174053059 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115205267 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚269684732455936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 174032995 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚242875998232576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 287645811 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚257336629189632 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174053059 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚242876016235520 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 287645811 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115205771 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12138213716992 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48881488 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 287665877 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884021815125117e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6806.149939617 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 113612816 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 287665877 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚243144977939470 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12138382730240 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 174053059 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115205267 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚269684732455936 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 174032995 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚242875998232576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 287645811 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚257336629189632 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 174053059 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚242876016235520 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 287645811 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115205771 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12138213716992 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚48881488Ão R2Ë!- _wandbruntime‚6806  _runtime‚6806Fáfá[Ì!Õ 0loss/FRt4/assignment_loss‚1.4783507597943146 1loss/SRqqt3/detection_loss‚1.8197730303091824 1loss/SRqqt4/detection_loss‚1.1693845282568307 epoch‚7 % _timestamp‚1.7884021815142958e+09 /loss/FRt1/assignment_loss‚2.560157710313797 /loss/FRt3/assignment_loss‚2.505618593096733 1loss/SRqqt1/assignment_loss‚1.356135993823409 /loss/FRt2/detection_loss‚2.8988082601178076 /loss/SRqqt2/detection_loss‚2.83579109787941 /loss/FBt1/detection_loss‚2.9006087243556977 _runtime‚ 6806.15103846 2loss/SRqqt4/assignment_loss‚0.7132879830591503 /loss/FBt3/detection_loss‚2.5075704672119836 0loss/FBt1/assignment_loss‚0.3597691263982597 /loss/FBt2/detection_loss‚2.9319789588451384 1loss/FBt4/assignment_loss‚0.19755837522459413 /loss/FRt1/detection_loss‚2.1340365424752235 %loss/total_loss‚42.48545806053377 0loss/FBt3/assignment_loss‚0.4560388966515761 1loss/SRqqt1/detection_loss‚1.7106350257992744 /loss/FBt4/detection_loss‚0.6130948524196568 trainer/global_step‚8599 0loss/FRt2/assignment_loss‚3.6292963720137075 1loss/FBt2/assignment_loss‚0.32710456833988244 2loss/SRqqt2/assignment_loss‚1.6326425668597222 2loss/SRqqt3/assignment_loss‚1.3592375588234302 /loss/FRt3/detection_loss‚2.4904636461287737 /loss/FRt4/detection_loss‚1.8981144223362207 _step‚708ÄÖç]`Í!Ú 1loss/FBt4/assignment_loss‚0.19755837522459413 /loss/FRt1/detection_loss‚2.1340365424752235 %loss/total_loss‚42.48545806053377 0loss/FBt3/assignment_loss‚0.4560388966515761 1loss/SRqqt1/detection_loss‚1.7106350257992744 /loss/FBt4/detection_loss‚0.6130948524196568 trainer/global_step‚8599 0loss/FRt2/assignment_loss‚3.6292963720137075 1loss/FBt2/assignment_loss‚0.32710456833988244 2loss/SRqqt2/assignment_loss‚1.6326425668597222 2loss/SRqqt3/assignment_loss‚1.3592375588234302 /loss/FRt3/detection_loss‚2.4904636461287737 /loss/FRt4/detection_loss‚1.8981144223362207 _step‚708 0loss/FRt4/assignment_loss‚1.4783507597943146 1loss/SRqqt3/detection_loss‚1.8197730303091824 1loss/SRqqt4/detection_loss‚1.1693845282568307 epoch‚7 % _timestamp‚1.7884021815142958e+09 /loss/FRt1/assignment_loss‚2.560157710313797 /loss/FRt3/assignment_loss‚2.505618593096733 1loss/SRqqt1/assignment_loss‚1.356135993823409 /loss/FRt2/detection_loss‚2.8988082601178076 /loss/SRqqt2/detection_loss‚2.83579109787941 /loss/FBt1/detection_loss‚2.9006087243556977 _runtime‚ 6806.15103846 2loss/SRqqt4/assignment_loss‚0.7132879830591503 /loss/FBt3/detection_loss‚2.5075704672119836 0loss/FBt1/assignment_loss‚0.3597691263982597 /loss/FBt2/detection_loss‚2.9319789588451384Ä´s4Î!‚(:… ‰´ãÔŽµ°® gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚138.04' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.608 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§m·Ï!‚(:ô ‰´ãÔï‘œ± network.recv‚ 2.409346e+06 disk.sda2.out‚0 disk.drbd2266.in‚0! disk.dm-0.out‚33803.455078125 disk.rbd1.in‚0& disk.nvme0c0n1.out‚33803.455078125 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.dm-0.in‚ 395.66015625 disk.sda1.out‚0 disk.nvme0n1.in‚ 395.66015625 cpu‚25.37475910815951* disk./.usagePercent‚15.581783112137362% disk./.usageGB‚232.17982864379883 disk.drbd2080.in‚0 disk.sda3.out‚ 7995.09765625$ disk.nvme0n1.out‚33803.455078125( proc.memory.availableMB‚ 655.35546875 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda.out‚ 7995.09765625! proc.memory.rssMB‚ 2685.171875* proc.memory.percent‚16.388988494873047 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.06640625 disk.dm-1.out‚ 7995.09765625 disk.drbd2266.out‚0$ memory_percent‚96.00002765655518 disk.rbd0.in‚0 disk.sda3.in‚ 0.06640625 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda.in‚ 0.06640625 proc.cpu.threads‚17 disk.rbd1.out‚ 16.5791015625" disk.nvme0c0n1.in‚ 395.66015625 network.sent‚ 2.128371e+07*ž8—‘Ð!‚(:† ˜´ãÔ¦ðû­ gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚139.685' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.937 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ð‚Ñ!‚(:÷ ˜´ãԟ𤰠disk.drbd1716.out‚0 disk.dm-1.in‚ 0.06640625 network.sent‚ 2.1392172e+07& disk.nvme0n1.out‚34285.46240234375 disk.sda2.out‚0 disk.drbd2266.in‚0# disk.dm-0.out‚34285.46240234375 disk.sda1.in‚0 disk.drbd2080.in‚0 disk.rbd1.in‚0! proc.memory.rssMB‚ 2685.171875 proc.cpu.threads‚17* disk./.usagePercent‚15.670711681283809 disk.drbd2080.out‚0" disk.nvme0c0n1.in‚ 396.05859375 cpu‚25.374018726760433 disk.dm-1.out‚ 7995.6484375 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 396.05859375 disk.sda1.out‚0 network.recv‚ 2.41511e+06 disk.sda3.out‚ 7995.6484375 disk.rbd1.out‚ 16.5791015625( disk.nvme0c0n1.out‚34285.46240234375* proc.memory.percent‚16.388988494873047$ disk./.usageGB‚233.5049285888672 disk.nvme0n1.in‚ 396.05859375 disk.sda.in‚ 0.06640625 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.sda3.in‚ 0.06640625$ memory_percent‚95.99874019622803( proc.memory.availableMB‚ 655.56640625 disk.sda.out‚ 7995.6484375 disk.sda2.in‚0 disk.rbd0.in‚0á• ûÒ!‚(:’ §´ãÔùƒ½® gpu.0.gpu‚24.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚143.469' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.693799999999996 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<ÿ1»|Ó!‚(:ñ §´ãÔåÛ—± disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd2266.in‚0$ memory_percent‚95.99871635437012 disk.dm-0.in‚ 399.828125 disk.sda1.out‚0& disk.nvme0n1.out‚35164.12060546875 disk.sda.in‚ 0.06640625 disk.sda.out‚ 7996.3515625 disk.dm-1.out‚ 7996.3515625! proc.memory.rssMB‚ 2685.171875 network.sent‚ 2.139851e+07 disk.nvme0c0n1.in‚ 399.828125 disk.nvme0n1.in‚ 399.828125 disk.sda2.in‚0 disk.dm-1.in‚ 0.06640625' proc.memory.availableMB‚ 655.5703125 disk.drbd1716.out‚0# disk.dm-0.out‚35164.12060546875 disk.sda1.in‚0 disk.sda3.in‚ 0.06640625 disk.rbd1.in‚0 disk.rbd1.out‚ 16.6416015625 disk.drbd2080.out‚0 disk.rbd0.in‚0* proc.memory.percent‚16.388988494873047 disk.drbd2080.in‚0 network.recv‚ 2.419597e+06% disk./.usageGB‚233.50660705566406( disk.nvme0c0n1.out‚35164.12060546875 disk.drbd2265.in‚0 disk.sda3.out‚ 7996.3515625 cpu‚25.373303759973993* disk./.usagePercent‚15.670824324598875 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 proc.cpu.threads‚17xŒx2Ô!- _wandbruntime‚6841  _runtime‚6841ž¬ÐÇÕ!Á & lr-AdamW/pg1‚4.882727487420954e-05 & lr-AdamW/pg2‚4.882727487420954e-05 trainer/global_step‚8649 % _timestamp‚1.7884022165688052e+09 _runtime‚6841.205638604 _step‚709ž%»ÜÌÖ!Æ & lr-AdamW/pg1‚4.882727487420954e-05 & lr-AdamW/pg2‚4.882727487420954e-05 trainer/global_step‚8649 % _timestamp‚1.7884022165688052e+09 _runtime‚6841.205638604 _step‚709Å û22×!- _wandbruntime‚6841  _runtime‚6841,›œhí'Ø!çO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 289239017 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚232312972297216 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12171469351986 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚271170722681344 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚256418689006080 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚244212197104640 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 289259057 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚66685756 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49147430 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 114233109 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚258753363652608 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚8649 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12206491901440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12206660901376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚6841.206493006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66685754 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚244212215107584 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12206660901376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚710 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 115832646 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114233107 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114233107 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚244212197104640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175005910 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚271170955108864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚256418876008960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚244484458688562 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 175025948 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12417359028736 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚256418876008960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12206491901440 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 % _timestamp‚1.7884022165694208e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚256418689006080 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12206660901376 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _runtime‚6841.206493006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚66685754 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚244212215107584 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12206660901376 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚710 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 115832646 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114233107 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114233107 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚244212197104640 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 175005910 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚271170955108864 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚256418876008960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚244484458688562 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 175025948 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12417359028736 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚256418876008960 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12206491901440 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 % _timestamp‚1.7884022165694208e+09 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚256418689006080 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289291529 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6842.107284212 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49152334 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12207885039616 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚244511608874678 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚232338932518912 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 trainer/global_step‚8649 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175025697 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12208054052864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289271463 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175025697 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚256447351508992 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12208054052864 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12172693395126 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚271201289519104 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚258782300540928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66693133 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚244511791035650 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115845996 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115845465 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12207885039616 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114245768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884022174697542e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12172858516738 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175045761 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚256447538525184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚244239484472320 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66693131 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 289271463 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚244239466469376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 289291529 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12418774212608 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 289291529 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚6842.107284212 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49152334 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12207885039616 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚244511608874678 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚232338932518912 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 trainer/global_step‚8649 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 175025697 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12208054052864 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 289271463 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 175025697 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚256447351508992 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12208054052864 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12172693395126 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚271201289519104 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚258782300540928 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚66693133 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚244511791035650 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 115845996 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 115845465 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12207885039616 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114245768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884022174697542e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12172858516738 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 175045761 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚256447538525184 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚244239484472320 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚66693131 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 289271463 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160ÇÒ’`2Ý!- _wandbruntime‚6842  _runtime‚6842nÚªZÞ!Ô /loss/FRt3/assignment_loss‚2.997623963488473 2loss/SRqqt2/assignment_loss‚1.8160292731993126 /loss/SRqqt3/assignment_loss‚1.3775573888407 1loss/SRqqt2/detection_loss‚2.4035451863751267 /loss/FBt3/detection_loss‚2.2066036740938824 0loss/FRt1/assignment_loss‚3.1736518857734546 2loss/SRqqt1/assignment_loss‚1.5075191137939692 /loss/FRt4/assignment_loss‚1.721740611319272 /loss/FRt4/detection_loss‚1.7256281291538815 0loss/SRqqt3/detection_loss‚1.678819421889647 /loss/FBt1/detection_loss‚2.2676407363679676 %loss/total_loss‚43.94057167916728 /loss/FRt2/assignment_loss‚4.198505851355466 % _timestamp‚1.7884022174718702e+09 2loss/SRqqt4/assignment_loss‚0.9823140836365166 0loss/FBt3/assignment_loss‚0.6012075023104709 /loss/FRt1/detection_loss‚2.1474877100970065 1loss/SRqqt4/detection_loss‚1.1095151110064416 epoch‚7 _step‚712 1loss/FBt1/assignment_loss‚0.30172704539735506 1loss/SRqqt1/detection_loss‚1.7755883288383485 /loss/FBt2/detection_loss‚2.8801667605127608 trainer/global_step‚8649 1loss/FBt2/assignment_loss‚0.36560519697981625 /loss/FRt3/detection_loss‚2.3887224752042027 _runtime‚6842.108605375 1loss/FBt4/assignment_loss‚0.18879009423556625 .loss/FRt2/detection_loss‚3.496758460998535 /loss/FBt4/detection_loss‚0.6278236742991095—®5Ð_ß!Ù 1loss/FBt1/assignment_loss‚0.30172704539735506 1loss/SRqqt1/detection_loss‚1.7755883288383485 /loss/FBt2/detection_loss‚2.8801667605127608 trainer/global_step‚8649 1loss/FBt2/assignment_loss‚0.36560519697981625 /loss/FRt3/detection_loss‚2.3887224752042027 _runtime‚6842.108605375 1loss/FBt4/assignment_loss‚0.18879009423556625 .loss/FRt2/detection_loss‚3.496758460998535 /loss/FBt4/detection_loss‚0.6278236742991095 /loss/FRt3/assignment_loss‚2.997623963488473 2loss/SRqqt2/assignment_loss‚1.8160292731993126 /loss/SRqqt3/assignment_loss‚1.3775573888407 1loss/SRqqt2/detection_loss‚2.4035451863751267 /loss/FBt3/detection_loss‚2.2066036740938824 0loss/FRt1/assignment_loss‚3.1736518857734546 2loss/SRqqt1/assignment_loss‚1.5075191137939692 /loss/FRt4/assignment_loss‚1.721740611319272 /loss/FRt4/detection_loss‚1.7256281291538815 0loss/SRqqt3/detection_loss‚1.678819421889647 /loss/FBt1/detection_loss‚2.2676407363679676 %loss/total_loss‚43.94057167916728 /loss/FRt2/assignment_loss‚4.198505851355466 % _timestamp‚1.7884022174718702e+09 2loss/SRqqt4/assignment_loss‚0.9823140836365166 0loss/FBt3/assignment_loss‚0.6012075023104709 /loss/FRt1/detection_loss‚2.1474877100970065 1loss/SRqqt4/detection_loss‚1.1095151110064416 epoch‚7 _step‚712Èsõ±’à!‚(:‡ ¶´ãÔ’¨à¯ gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚144.287' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.8574 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Qv{á!‚(:ð ¶´ãÔý¾×± cpu‚25.372558788252036 network.sent‚ 2.1404004e+07 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 399.828125! proc.memory.rssMB‚ 2685.171875* proc.memory.percent‚16.388988494873047 proc.cpu.threads‚17 disk.sda.out‚ 7996.7578125 disk.sda2.out‚0# disk.dm-0.out‚35164.94970703125 disk.sda3.out‚ 7996.7578125 network.recv‚ 2.423241e+06( disk./.usagePercent‚15.6708125482523 disk.dm-1.in‚ 0.06640625 disk.drbd2080.out‚0( disk.nvme0c0n1.out‚35164.94970703125 disk.dm-1.out‚ 7996.7578125 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 16.6416015625& disk.nvme0n1.out‚35164.94970703125 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.rbd0.in‚0# memory_percent‚95.9986925125122 disk.rbd1.in‚0 disk.sda.in‚ 0.06640625 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 399.828125% disk./.usageGB‚233.50643157958984 disk.dm-0.in‚ 399.828125 disk.sda3.in‚ 0.06640625( proc.memory.availableMB‚ 655.57421875>f>t’â!‚(:‡ Å´ãÔß®™® gpu.0.gpu‚42.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚123.751' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7502 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ý;D|ã!‚(:ñ Å´ãÔ©äØ°& proc.memory.availableMB‚ 655.546875 disk.sda1.out‚0 disk.sda.out‚ 7997.44140625 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 399.828125 network.recv‚ 2.429684e+06) disk./.usagePercent‚15.67078054731052 disk.sda3.in‚ 0.06640625 disk.sda3.out‚ 7997.44140625( disk.nvme0c0n1.out‚35165.30517578125 disk.sda.in‚ 0.06640625 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 7997.44140625 disk.drbd2266.in‚0 disk.drbd2080.out‚0* proc.memory.percent‚16.388988494873047 cpu‚25.37182832183486 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 399.828125 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.06640625# disk.dm-0.out‚35165.30517578125! proc.memory.rssMB‚ 2685.171875 network.sent‚ 2.1513016e+07% disk./.usageGB‚233.50595474243164 disk.sda1.in‚0 disk.rbd1.out‚ 16.767578125 disk.rbd0.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17& disk.nvme0n1.out‚35165.30517578125 disk.nvme0n1.in‚ 399.828125$ memory_percent‚95.99885940551758~O72ä!- _wandbruntime‚6877  _runtime‚6877”µÇå!Á & lr-AdamW/pg1‚4.753312962643262e-05 & lr-AdamW/pg2‚4.753312962643262e-05 trainer/global_step‚8699 % _timestamp‚1.7884022524738092e+09 _runtime‚6877.110636895 _step‚713ê´oÌæ!Æ trainer/global_step‚8699 % _timestamp‚1.7884022524738092e+09 _runtime‚6877.110636895 _step‚713 & lr-AdamW/pg1‚4.753312962643262e-05 & lr-AdamW/pg2‚4.753312962643262e-05É pÖ}2ç!- _wandbruntime‚6877  _runtime‚6877›5>‡Žè!çO KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49417910 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚272687311272960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 290884709 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12488300209152 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚233610131414016 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚257851828565504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 116472514 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114866057 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114866059 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114866057 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176018650 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚245575683344384 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12240941170394 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12276332224000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 290864669 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12276163224064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚260199035003904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚257851828565504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 175998612 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚245851089623770 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚245575665341440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚257852015568384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67054606 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49418420 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚245850907463006 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚245575665341440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272687543700480 % _timestamp‚1.7884022524745011e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚260199011063808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚714 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚245575683344384 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0!`7ò'é!ìO RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49417910 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚272687311272960 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 290884709 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12488300209152 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚233610131414016 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚257851828565504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 116472514 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 114866057 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 114866059 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 114866057 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 176018650 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚245575683344384 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12240941170394 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12276332224000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 290864669 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12276163224064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚260199035003904 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚257851828565504 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 175998612 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚245851089623770 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚245575665341440 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚257852015568384 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67054606 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49418420 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚245850907463006 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚245575665341440 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚272687543700480 % _timestamp‚1.7884022524745011e+09 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚260199011063808 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 _step‚714 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚245575683344384 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520ʈ„š2ê!- _wandbruntime‚6877  _runtime‚6877½àŒj÷&ë!ñM OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114878718 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _step‚715 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚245878421970858 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚257880678084608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884022531437385e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49424080 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49423567 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚260227924012032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114878718 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67061983 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12489929222656 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚257880491068416 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 290917181 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚245602952709120 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚233636074596352 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 176018399 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚245602934706176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 290897115 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚272717877174784 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8699 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 176018399 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116485548 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 116486063 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 290917181 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12242330335146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12277725375488 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚245602952709120 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114878716 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _runtime‚6877.780934968 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12277556362240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚260227947952128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 290897115 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12242165213534 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚515 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12489712359936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚245878239809886 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969|\‚Ècì!öM B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 176018399 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 116485548 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 116486063 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 290917181 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12242330335146 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12277725375488 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚245602952709120 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 114878716 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _runtime‚6877.780934968 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12277556362240 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚260227947952128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 290897115 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12242165213534 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚515 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12489712359936 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚245878239809886 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 114878718 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _step‚715 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚245878421970858 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚257880678084608 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884022531437385e+09 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49424080 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 216862720 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49423567 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚260227924012032 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 114878718 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67061983 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12489929222656 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.oä¡ê™ n_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚257880491068416 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 290917181 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚245602952709120 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚233636074596352 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 176018399 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚245602934706176 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 290897115 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚272717877174784 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚8699 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 176991314 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚261644682415104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12346003546624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177011352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚8749 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚259285155127808 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12345834546688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 292510361 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259284968124928 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12345834546688 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚274203845128704 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚246939151581184 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12559403529728 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117113440 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12310412988802 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12559186653696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115499009 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚247217538398214 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67423456 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚246939151581184 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115499007 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚541 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 292490321 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259285155127808 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12346003546624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49689984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115499009 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚234907307570176 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚259284968124928 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚6913.276702441 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚247217720558978 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115499007 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚246939133578240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚274204085944832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 $ _timestamp‚1.788402288639574e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚261644658475008 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 176991314 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚261644682415104 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12346003546624 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 177011352 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚8749 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚259285155127808 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12345834546688 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 292510361 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚259284968124928 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12345834546688 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚274203845128704 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚246939151581184 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12559403529728 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117113440 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12310412988802 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12559186653696 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 115499009 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚247217538398214 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚67423456 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚246939151581184 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 115499007 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚541 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 292490321 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚259285155127808 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12346003546624 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49689984 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 115499009 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚234907307570176 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚259284968124928 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 _runtime‚6913.276702441 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚247217720558978 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 115499007 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚543 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚246939133578240 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚274204085944832 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 $ _timestamp‚1.788402288639574e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚261644658475008 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 JDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117125606 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/ac-Outive.all.allocated‚ 292542833 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚261673571423232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚246966402942976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12347396698112 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292542833 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12347227684864 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49695296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221928448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚274234410460672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚234933233713152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12347396698112 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115511666 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚247245052906066 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 177031165 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49694775 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115511668 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚719 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 292522767ôØXŸû&ü!õM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 292542833 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚261673571423232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚246966402942976 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12347396698112 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 292542833 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12347227684864 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49695296 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221928448 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚274234410460672 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚234933233713152 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12347396698112 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 115511666 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚247245052906066 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 177031165 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49694775 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 115511668 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚719 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 292522767 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 292522767 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚67430833 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 197988352 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚234933250752512 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚246966420945920 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12347227684864 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 115511668 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 $ _timestamp‚1.788402289312122e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 177011101 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 trainer/global_step‚8749 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67430831 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117125606 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ÏùB~2ý!- _wandbruntime‚6913  _runtime‚6913ëVþ!Ð %loss/total_loss‚43.60035257326647 epoch‚7 trainer/global_step‚8749 /loss/FBt1/detection_loss‚2.3352743731095242 0loss/FRt1/assignment_loss‚2.8270401766425683 0loss/SRqqt2/detection_loss‚2.505072075341429 _runtime‚6913.950579707 0loss/FRt2/assignment_loss‚3.2975446031643796 /loss/FBt4/detection_loss‚0.5706026983362708 1loss/SRqqt4/assignment_loss‚0.861253088208991 0loss/FBt1/assignment_loss‚0.4212210418863833 .loss/FRt2/detection_loss‚3.315254793717311 _step‚720 /loss/FRt1/detection_loss‚2.0721875643521463 .loss/FRt3/detection_loss‚3.008519333600998 /loss/FRt3/assignment_loss‚2.641667171319326 0loss/FRt4/assignment_loss‚1.4159909248352052 1loss/SRqqt1/assignment_loss‚1.421958939731121 % _timestamp‚1.7884022893138275e+09 1loss/SRqqt2/assignment_loss‚1.770682123622724 2loss/SRqqt3/assignment_loss‚1.7283560631530626 0loss/FBt2/assignment_loss‚0.2533753202720206 1loss/FBt4/assignment_loss‚0.10908095090867773 .loss/FRt4/detection_loss‚1.561452366709709 /loss/SRqqt1/detection_loss‚1.84832110196352 1loss/SRqqt4/detection_loss‚1.0362978555576903 /loss/FBt2/detection_loss‚3.1263513697518244 0loss/FBt3/assignment_loss‚0.1323574452516354 1loss/SRqqt3/detection_loss‚2.3297242982046944 /loss/FBt3/detection_loss‚3.0107668936252594é9âT[ÿ!Õ /loss/FRt1/detection_loss‚2.0721875643521463 .loss/FRt3/detection_loss‚3.008519333600998 /loss/FRt3/assignment_loss‚2.641667171319326 0loss/FRt4/assignment_loss‚1.4159909248352052 1loss/SRqqt1/assignment_loss‚1.421958939731121 % _timestamp‚1.7884022893138275e+09 1loss/SRqqt2/assignment_loss‚1.770682123622724 2loss/SRqqt3/assignment_loss‚1.7283560631530626 0loss/FBt2/assignment_loss‚0.2533753202720206 1loss/FBt4/assignment_loss‚0.10908095090867773 .loss/FRt4/detection_loss‚1.561452366709709 /loss/SRqqt1/detection_loss‚1.84832110196352 1loss/SRqqt4/detection_loss‚1.0362978555576903 /loss/FBt2/detection_loss‚3.1263513697518244 0loss/FBt3/assignment_loss‚0.1323574452516354 1loss/SRqqt3/detection_loss‚2.3297242982046944 /loss/FBt3/detection_loss‚3.0107668936252594 %loss/total_loss‚43.60035257326647 epoch‚7 trainer/global_step‚8749 /loss/FBt1/detection_loss‚2.3352743731095242 0loss/FRt1/assignment_loss‚2.8270401766425683 0loss/SRqqt2/detection_loss‚2.505072075341429 _runtime‚6913.950579707 0loss/FRt2/assignment_loss‚3.2975446031643796 /loss/FBt4/detection_loss‚0.5706026983362708 1loss/SRqqt4/assignment_loss‚0.861253088208991 0loss/FBt1/assignment_loss‚0.4212210418863833 .loss/FRt2/detection_loss‚3.315254793717311 _step‚720ÐêÙ0a€"‚(:’ ò´ãԷ̨® gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚101.377' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.275399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0XñÊ€"‚(:õ ò´ãÔàÁ³° disk.drbd2265.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 7999.46484375( proc.memory.availableMB‚ 655.00390625* proc.memory.percent‚16.388988494873047 proc.cpu.threads‚17 disk.drbd2266.in‚0 disk.rbd1.out‚ 16.830078125& disk.nvme0n1.out‚35168.21826171875 disk.sda.in‚ 0.06640625 disk.dm-1.in‚ 0.06640625 disk.rbd0.out‚0 disk.dm-0.in‚ 399.828125 disk.sda3.in‚ 0.06640625 disk.sda2.in‚0 disk.dm-1.out‚ 7999.46484375 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 399.828125 cpu‚25.369671702016635( disk.nvme0c0n1.out‚35168.21826171875$ memory_percent‚96.00217342376709% disk./.usageGB‚233.50687789916992 disk.rbd0.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 399.828125! proc.memory.rssMB‚ 2685.171875 network.recv‚ 2.443596e+06 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda.out‚ 7999.46484375 disk.sda2.out‚0 network.sent‚ 2.1633194e+07* disk./.usagePercent‚15.670842501133805 disk.drbd2266.out‚0# disk.dm-0.out‚35168.21826171875 disk.drbd1716.in‚0Š¡ßM’‚"‚(:‡ µãÔ«¨²® gpu.0.gpu‚24.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚133.486' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.6972 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Œ¦pªrƒ"‚(:ç µãÔ–Ѐ±* disk./.usagePercent‚15.670846085239285 disk.drbd1716.out‚0 disk.dm-1.out‚ 7999.859375 disk.rbd0.in‚0$ disk.nvme0n1.out‚35168.486328125 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.dm-0.in‚ 399.828125 disk.sda3.in‚ 0.06640625 network.recv‚ 2.449735e+06 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda3.out‚ 7999.859375 disk.nvme0c0n1.in‚ 399.828125& disk.nvme0c0n1.out‚35168.486328125 disk.drbd1716.in‚0 disk.rbd1.out‚ 17.0419921875( proc.memory.availableMB‚ 655.07421875! proc.memory.rssMB‚ 2685.171875% disk./.usageGB‚233.50693130493164 disk.nvme0n1.in‚ 399.828125 disk.sda.out‚ 7999.859375 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.06640625! disk.dm-0.out‚35168.486328125 disk.sda1.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0* proc.memory.percent‚16.388988494873047 cpu‚25.36901359641046 disk.sda.in‚ 0.06640625# memory_percent‚96.0017442703247 proc.cpu.threads‚17 network.sent‚ 2.174177e+07RØÀô‘„"‚(:† µãÔ¦¾£® gpu.0.gpu‚9.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚124.046' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.8092 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0à+÷ u…"‚(:ê µãÔɾ‘°* disk./.usagePercent‚15.670679936349563 disk.drbd2080.out‚0 disk.dm-0.out‚ 35169.5234375" disk.nvme0n1.out‚ 35169.5234375 disk.sda.in‚ 0.06640625 disk.dm-1.out‚ 8000.74609375* proc.memory.percent‚16.388988494873047 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda3.out‚ 8000.74609375 disk.drbd1716.in‚0 disk.drbd1716.out‚0( proc.memory.availableMB‚ 654.95703125 disk.sda1.out‚0$ disk.nvme0c0n1.out‚ 35169.5234375 disk.nvme0n1.in‚ 399.828125 disk.sda2.out‚0! proc.memory.rssMB‚ 2685.171875 cpu‚25.368297793389228 disk.dm-1.in‚ 0.06640625 network.sent‚ 2.1747557e+07% disk./.usageGB‚233.50445556640625 disk.rbd0.out‚0 disk.rbd1.in‚0 network.recv‚ 2.453213e+06 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚ 0.06640625 disk.nvme0c0n1.in‚ 399.828125 disk.sda.out‚ 8000.74609375 disk.sda2.in‚0 disk.drbd2266.in‚0 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.dm-0.in‚ 399.828125 disk.rbd1.out‚ 17.0419921875$ memory_percent‚96.00245952606201–Çû2†"- _wandbruntime‚6948  _runtime‚6948ÛšŠ^LJ"Á trainer/global_step‚8799 % _timestamp‚1.7884023242339113e+09 _runtime‚6948.870771843 _step‚721 & lr-AdamW/pg1‚4.498259305791211e-05 & lr-AdamW/pg2‚4.498259305791211e-05Gxäø̈"Æ _step‚721 & lr-AdamW/pg1‚4.498259305791211e-05 & lr-AdamW/pg2‚4.498259305791211e-05 trainer/global_step‚8799 % _timestamp‚1.7884023242339113e+09 _runtime‚6948.870771843Ñs±J@2‰"- _wandbruntime‚6948  _runtime‚6948 ÷Š"çO LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116131959 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12415674869248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _runtime‚6948.871662513 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚236204449647616 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚248302619817984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67792304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚260718294687232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚260718294687232 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚260718107684352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12630041111552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚248302601815040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116131959 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116131957 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚248302601815040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚263090305886208 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚260718107684352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117753373 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12379884807210 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 % _timestamp‚1.7884023242346282e+09 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177984016 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117752841 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49960537 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49961067 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚275720346997760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178004054 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚722 QCDeviceStatsMonitor.Dþ¼n on_train_batch_start/allocation.large_pool.freed‚ 116131957 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚236204466686976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 294115973 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1]Âãò'‹"ìO G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚248302619817984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚67792304 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚260718294687232 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚260718294687232 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚260718107684352 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12630041111552 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚248302601815040 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 116131959 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116131957 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚248302601815040 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚263090305886208 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚260718107684352 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 117753373 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12379884807210 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 % _timestamp‚1.7884023242346282e+09 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 177984016 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 117752841 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚49960537 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚49961067 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚275720346997760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178004054 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚722 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116131957 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚236204466686976 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 294115973 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116131959 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12415674869248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _runtime‚6948.871662513 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚236204449647616Ò?#2Œ"- _wandbruntime‚6949  _runtime‚6949½Y³4÷&"ñM E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚67799681 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚505 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294168485 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178023867 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚248611501680302 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116144616 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 _runtime‚6949.537806822 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚275750676454912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12416899007488 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚49966327 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚248329889182720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 trainer/global_step‚8799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117766008 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 _step‚723 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚275750910966272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚260746770187264 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 294148419 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116144618 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚260746957203456 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116144618 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚503 % _timestamp‚1.7884023249004288e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 117766513 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49966830 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚248611683841274 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚248329889182720 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12417068020736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12381108850350 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 178023867 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12631457620480 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 294168485 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚260746957203456 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚248329871179776 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚248329889182720 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 trainer/global_step‚8799 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 117766008 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 _step‚723 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚275750910966272 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚260746770187264 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 294148419 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116144618 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚260746957203456 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116144618 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚503 % _timestamp‚1.7884023249004288e+09 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 117766513 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚49966830 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚248611683841274 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚248329889182720 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12417068020736 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12381108850350 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 178023867 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12631457620480 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 294168485 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚260746957203456 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚248329871179776 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚505 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 294168485 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 178023867 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚248611501680302 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 116144616 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 _runtime‚6949.537806822 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚275750676454912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0ÓKTÇ2"- _wandbruntime‚6949  _runtime‚6949ÖŸ"× 0loss/FRt2/assi‰kð4Egnment_loss‚4.3928879431315835 0loss/FRt3/assignment_loss‚2.4747878655791284 .loss/FBt3/detection_loss‚2.606277310848236 trainer/global_step‚8799 2loss/SRqqt2/assignment_loss‚2.0730368455932027 1loss/FBt4/assignment_loss‚0.14974817211249558 1loss/SRqqt3/detection_loss‚1.5059139370918273 /loss/FBt1/detection_loss‚1.9690312309698625 /loss/FBt2/detection_loss‚3.9217926348958696 1loss/SRqqt1/assignment_loss‚1.681833226005642 /loss/FBt4/detection_loss‚0.7510690853491219 _runtime‚6949.538964466 _step‚724 0loss/FRt1/assignment_loss‚2.9212169528007506 0loss/FBt1/assignment_loss‚0.3245483938578259 1loss/SRqqt1/detection_loss‚1.6911112629637426 $loss/total_loss‚43.9574493762207 epoch‚7 1loss/FBt2/assignment_loss‚0.18528664103617334 /loss/FRt2/detection_loss‚2.7855848363467626 /loss/FRt3/detection_loss‚2.2502591852098703 1loss/SRqqt4/detection_loss‚1.1562021327753589 /loss/FRt1/detection_loss‚2.1967564198705887 /loss/FRt4/detection_loss‚1.7521590388246946 0loss/SRqqt2/detection_loss‚2.729197809773107 0loss/FRt4/assignment_loss‚1.5240686633148972 2loss/SRqqt4/assignment_loss‚0.9338269634204215 1loss/FBt3/assignment_loss‚0.30801240652799605 1loss/SRqqt3/assignment_loss‚1.672840417921543 % _timestamp‚1.7884023249022171e+09༠†b‘"Ü 1loss/FBt3/assignment_loss‚0.30801240652799605 1loss/SRqqt3/assignment_loss‚1.672840417921543 % _timestamp‚1.7884023249022171e+09 0loss/FRt2/assignment_loss‚4.3928879431315835 0loss/FRt3/assignment_loss‚2.4747878655791284 .loss/FBt3/detection_loss‚2.606277310848236 trainer/global_step‚8799 2loss/SRqqt2/assignment_loss‚2.0730368455932027 1loss/FBt4/assignment_loss‚0.14974817211249558 1loss/SRqqt3/detection_loss‚1.5059139370918273 /loss/FBt1/detection_loss‚1.9690312309698625 /loss/FBt2/detection_loss‚3.9217926348958696 1loss/SRqqt1/assignment_loss‚1.681833226005642 /loss/FBt4/detection_loss‚0.7510690853491219 _runtime‚6949.538964466 _step‚724 0loss/FRt1/assignment_loss‚2.9212169528007506 0loss/FBt1/assignment_loss‚0.3245483938578259 1loss/SRqqt1/detection_loss‚1.6911112629637426 $loss/total_loss‚43.9574493762207 epoch‚7 1loss/FBt2/assignment_loss‚0.18528664103617334 /loss/FRt2/detection_loss‚2.7855848363467626 /loss/FRt3/detection_loss‚2.2502591852098703 1loss/SRqqt4/detection_loss‚1.1562021327753589 /loss/FRt1/detection_loss‚2.1967564198705887 /loss/FRt4/detection_loss‚1.7521590388246946 0loss/SRqqt2/detection_loss‚2.729197809773107 0loss/FRt4/assignment_loss‚1.5240686633148972 2loss/SRqqt4/assignment_loss‚0.9338269634204215Ô{ cÞ’"‚(:’ ŸµãԨї® gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚134.841' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.968200000000003 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Û‡5|w“"‚(:ì ŸµãÔ¸¸á° disk.sda.out‚ 8001.2109375 disk.rbd1.in‚0' disk.nvme0c0n1.out‚35169.9267578125 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0" disk.dm-0.out‚35169.9267578125! proc.memory.rssMB‚ 2685.171875 disk.sda3.in‚ 0.06640625 disk.sda3.out‚ 8001.2109375 disk.dm-1.out‚ 8001.2109375 disk.drbd2080.in‚0 disk.drbd2080.out‚0 proc.cpu.threads‚17% disk.nvme0n1.out‚35169.9267578125% disk./.usageGB‚233.50515747070312 disk.rbd1.out‚ 17.1044921875 disk.nvme0n1.in‚ 399.828125 disk.dm-1.in‚ 0.06640625 disk.rbd0.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0$ proc.memory.availableMB‚654.9375* proc.memory.percent‚16.388988494873047* disk./.usagePercent‚15.670727041735862 disk.sda.in‚ 0.06640625 disk.rbd0.out‚0 disk.sda1.out‚0 network.sent‚ 2.1753212e+07 disk.nvme0c0n1.in‚ 399.828125 disk.sda1.in‚0 network.recv‚ 2.457669e+06 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.dm-0.in‚ 399.828125$ memory_percent‚96.00257873535156 cpu‚25.367609563302363q'–¾‘”"‚(:† ®µãÔó騮 gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚139.155' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.831 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨â¨x•"‚(:í ®µãÔøó—° disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.sda3.in‚ 0.06640625 disk.nvme0n1.in‚ 399.828125 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.06640625 cpu‚25.3669332856382 network.recv‚ 2.4633e+06 disk.drbd2080.in‚0 disk.rbd1.out‚ 17.1044921875 disk.nvme0c0n1.in‚ 399.828125 disk.sda2.in‚0 disk.drbd2266.out‚0' disk.nvme0c0n1.out‚35170.5771484375* proc.memory.percent‚16.388988494873047% disk./.usageGB‚233.50519943237305 disk.drbd2265.out‚0 disk.rbd0.in‚0) disk./.usagePercent‚15.67072985781874 disk.rbd0.out‚0 disk.sda3.out‚ 8001.87890625 disk.rbd1.in‚0 disk.sda.in‚ 0.06640625' proc.memory.availableMB‚ 655.1328125 network.sent‚ 2.1861376e+07 disk.sda1.in‚0 disk.sda.out‚ 8001.87890625$ memory_percent‚96.00138664245605 disk.dm-0.in‚ 399.828125" disk.dm-0.out‚35170.5771484375 disk.sda1.out‚0 disk.sda2.out‚0% disk.nvme0n1.out‚35170.5771484375 disk.drbd1716.in‚0 disk.dm-1.out‚ 8001.87890625! proc.memory.rssMB‚ 2685.171875 proc.cpu.threads‚17êòF"2–"- _wandbruntime‚6984  _runtime‚6984Ö¡‚[É—"à ' lr-AdamW/pg1‚4.3726733265166605e-05 ' lr-AdamW/pg2‚4.3726733265166605e-05 trainer/global_step‚8849 % _timestamp‚1.7884023600308826e+09 _runtime‚6984.667757403 _step‚7259‘&¾Î˜"È trainer/global_step‚8849 % _timestamp‚1.7884023600308826e+09 _runtime‚6984.667757403 _step‚725 ' lr-AdamW/pg1‚4.3726733265166605e-05 ' lr-AdamW/pg2‚4.3726733265166605e-05Õ^Ë™2™"- _wandbruntime‚6984  _runtime‚6984a¦ ãí'š"çO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68161154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116764909 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12485177191936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚237501625803776 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118392765 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178996756 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚277237151653376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68161156 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 _runtime‚6984.668673372 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178996756 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚249666088054784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12485177191936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50231072 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 178976718 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚249666070051840 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116764907 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178976718 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚264535977237504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12700959636992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116764907 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚262151247243776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12485346191872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118392226 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12701174415872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12485346191872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚249666088054784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 % _timestamp‚1.7884023600316076e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚537 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚22‰û‚ò'›"ìO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68161156 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 _runtime‚6984.668673372 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 178996756 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚249666088054784 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12485177191936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50231072 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 178976718 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚249666070051840 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 116764907 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 178976718 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚264535977237504 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12700959636992 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 116764907 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚262151247243776 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12485346191872 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 118392226 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12701174415872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12485346191872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚249666088054784 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 % _timestamp‚1.7884023600316076e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚537 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68161154 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 116764909 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12485177191936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚237501625803776 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 118392765 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 178996756 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚277237151653376 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0ÖCÌT2œ"- _wandbruntime‚6985  _runtime‚6985W|žrÎ"ñM ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12450745790370 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚249693339416576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50236914 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12450580668758 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68168533 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118404951 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚249978132615510 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68168531 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202182656 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12486570330112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 295774071 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179016569 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 1´R)SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12702586555392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12486570330112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚6985.337224336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚237527568986112 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 % _timestamp‚1.7884023606998873e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚249693357419520 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚237527551946752 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116777568 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 295794137 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116777568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚727 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12486739343360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚262180096762880 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚264564890185728 trainer/global_step‚8849 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178996505 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚262180096762880 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116777566 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚249978314776482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 295794137 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226122752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚264564866245632 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 118404951 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚249978132615510 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68168531 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202182656 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12486570330112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 295774071 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 179016569 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12702586555392 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12486570330112 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚6985.337224336 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚237527568986112 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 % _timestamp‚1.7884023606998873e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚249693357419520 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚237527551946752 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 116777568 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 295794137 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 116777568 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 _step‚727 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12486739343360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚262180096762880 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚496 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚264564890185728 trainer/global_step‚8849 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 178996505 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚262180096762880 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 116777566 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚249978314776482 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 295794137 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226122752 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚264564866245632 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚249693339416576 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50236914 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12450580668758 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68168533 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0×7»™2Ÿ"- _wandbruntime‚6985  _runtime‚6985à~†ž] "× 0loss/SRqqt2/detection_loss‚2.904063736016934 0loss/FRt4/assignment_loss‚1.6625838119250078 2loss/SRqqt2/assignment_loss‚1.4626376614547694 2loss/SRqqt3/assignment_loss‚1.7171269339243216 2loss/SRqqt4/assignment_loss‚1.0009069869560854 trainer/global_step‚8849 1loss/SRqqt3/detection_loss‚1.8398237191140652 0loss/FRt3/assignment_loss‚2.8203465789556503 1loss/SRqqt4/detection_loss‚1.1129030514508487 /loss/FBt1/detection_loss‚1.8208886921405791 % _timestamp‚1.7884023607014632e+09 1loss/SRqqt1/detection_loss‚1.7208466794130937 /loss/FBt3/detection_loss‚2.5221008509397507 0loss/FRt1/assignment_loss‚3.2387609217987685 /loss/FRt2/assignment_loss‚3.488321139894683 1loss/FBt3/assignment_loss‚0.21746148273764732 /loss/FRt3/detection_loss‚2.5348105058073997 2loss/SRqqt1/assignment_loss‚1.5119152882329698 0loss/FBt2/assignment_loss‚0.5108735933898743 /loss/FRt1/detection_loss‚2.0531438496269163 -loss/FRt2/detection_loss‚2.98859481153817 /loss/FBt2/detection_loss‚3.0316657423973083 0loss/FBt1/assignment_loss‚0.6420252675419988 &loss/total_loss‚43.609131084111894 epoch‚7 _step‚728 0loss/FBt4/assignment_loss‚0.1511321754141398 /loss/FRt4/detection_loss‚1.8347248306068091 /loss/FBt4/detection_loss‚0.8214727728341061 _runtime‚6985.338213563ù.éb¡"Ü 2loss/SRqqt1/assignment_loss‚1.5119152882329698 0loss/FBt2/assignment_loss‚0.5108735933898743 /loss/FRt1/detection_loss‚2.0531438496269163 -loss/FRt2/detection_loss‚2.98859481153817 /loss/FBt2/detection_loss‚3.0316657423973083 0loss/FBt1/assignment_loss‚0.6420252675419988 &loss/total_loss‚43.609131084111894 epoch‚7 _step‚728 0loss/FBt4/assignment_loss‚0.1511321754141398 /loss/FRt4/detection_loss‚1.8347248306068091 /loss/FBt4/detection_loss‚0.8214727728341061 _runtime‚6985.338213563 0loss/SRqqt2/detection_loss‚2.904063736016934 0loss/FRt4/assignment_loss‚1.6625838119250078 2loss/SRqqt2/assignment_loss‚1.4626376614547694 2loss/SRqqt3/assignment_loss‚1.7171269339243216 2loss/SRqqt4/assignment_loss‚1.0009069869560854 trainer/global_step‚8849 1loss/SRqqt3/detection_loss‚1.8398237191140652 0loss/FRt3/assignment_loss‚2.8203465789556503 1loss/SRqqt4/detection_loss‚1.1129030514508487 /loss/FBt1/detection_loss‚1.8208886921405791 % _timestamp‚1.7884023607014632e+09 1loss/SRqqt1/detection_loss‚1.7208466794130937 /loss/FBt3/detection_loss‚2.5221008509397507 0loss/FRt1/assignment_loss‚3.2387609217987685 /loss/FRt2/assignment_loss‚3.488321139894683 1loss/FBt3/assignment_loss‚0.21746148273764732 /loss/FRt3/detection_loss‚2.5348105058073997ØJf7é‘¢"‚(:† ½µãԱǯ gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚121.625' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.325 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±®,x£"‚(:í ½µãÔ¯’è°! proc.memory.rssMB‚ 2685.171875% disk.nvme0n1.out‚35170.8076171875 disk.sda.in‚ 0.06640625 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.rbd0.out‚0" disk.dm-0.out‚35170.8076171875) disk./.usagePercent‚15.67073472196189 disk.sda1.out‚0 disk.sda3.in‚ 0.06640625 disk.sda.out‚ 8002.22265625* proc.memory.percent‚16.388988494873047 network.recv‚ 2.46756e+06$ disk./.usageGB‚233.5052719116211 disk.rbd1.out‚ 17.23046875 disk.sda2.in‚0 disk.drbd1716.in‚0' proc.memory.availableMB‚ 654.8515625' disk.nvme0c0n1.out‚35170.8076171875 disk.nvme0n1.in‚ 399.828125 disk.dm-1.in‚ 0.06640625 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 network.sent‚ 2.1867353e+07 disk.sda3.out‚ 8002.22265625 disk.nvme0c0n1.in‚ 399.828125 disk.dm-0.in‚ 399.828125$ memory_percent‚96.00310325622559 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.dm-1.out‚ 8002.22265625 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.sda1.in‚0 disk.sda2.out‚0 cpu‚25.366250822957788 disk.drbd2266.in‚0¸,¤"‚(:’ ̵ãÔðÛ‡® gpu.0.gpu‚36.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚143.469' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.693799999999996 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0sSÃq¥"‚(:æ ̵ãÔŸÄ° disk.sda3.out‚ 8002.734375$ memory_percent‚96.00310325622559 disk.nvme0c0n1.in‚ 399.828125 disk.nvme0n1.in‚ 399.828125 network.sent‚ 2.1975981e+07 network.recv‚ 2.473801e+06 disk.sda2.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0! disk.dm-0.out‚35171.322265625 disk.sda3.in‚ 0.06640625 disk.sda.in‚ 0.06640625 disk.drbd2080.in‚0! proc.memory.rssMB‚ 2685.171875 cpu‚25.365545838316386 disk.sda1.in‚0& disk.nvme0c0n1.out‚35171.322265625 disk.sda.out‚ 8002.734375 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2266.in‚0* disk./.usagePercent‚15.670736258007095$ disk./.usageGB‚233.5052947998047 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.dm-0.in‚ 399.828125' proc.memory.availableMB‚ 654.8515625 disk.rbd1.in‚0$ disk.nvme0n1.out‚35171.322265625 disk.sda2.in‚0 disk.rbd0.in‚0* proc.memory.percent‚16.388988494873047 disk.rbd1.out‚ 17.23046875 disk.dm-1.out‚ 8002.734375 proc.cpu.threads‚17 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.sda1.out‚0v¡u¦"‚(:’ ÛµãÔðÐŽ® gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚117.841' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.568199999999997 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0k£É#w§"‚(:ì ÛµãÔßé¹°) disk./.usagePercent‚15.67074009812011 disk.dm-1.in‚ 0.06640625 disk.rbd0.in‚0 disk.dm-0.in‚ 399.828125 disk.nvme0c0n1.in‚ 399.828125! proc.memory.rssMB‚ 2685.171875 disk.drbd2266.out‚0$ memory_percent‚96.00310325622559% disk./.usageGB‚233.50535202026367 disk.sda.in‚ 0.06640625 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.sda3.out‚ 8003.19140625' proc.memory.availableMB‚ 654.8515625 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0* proc.memory.percent‚16.388988494873047 disk.sda2.in‚0 disk.drbd1716.in‚0& disk.nvme0c0n1.out‚35171.537109375 network.sent‚ 2.1982365e+07 disk.drbd2080.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.06640625$ disk.nvme0n1.out‚35171.537109375 disk.dm-1.out‚ 8003.19140625 disk.drbd2265.out‚0! disk.dm-0.out‚35171.537109375 disk.rbd1.in‚0 disk.nvme0n1.in‚ 399.828125 proc.cpu.threads‚17 network.recv‚ 2.478837e+06 disk.sda.out‚ 8003.19140625 disk.rbd1.out‚ 17.29296875 cpu‚25.364871375162508å(”2¨"- _wandbruntime‚7020  _runtime‚7020©¦föÉ©"à trainer/global_step‚8899 % _timestamp‚1.7884023955294716e+09 _runtime‚7020.166312821 _step‚729 ' lr-AdamW/pg1‚4.2484162330774353e-05 ' lr-AdamW/pg2‚4.2484162330774353e-05»U¯‰Îª"È ' lr-AdamW/pg1‚4.2484162330774353e-05 ' lr-AdamW/pg2‚4.2484162330774353e-05 trainer/global_step‚8899 % _timestamp‚1.7884023955294716e+09 _runtime‚7020.166312821 _step‚729Ù‘_—Þ2«"- _wandbruntime‚7020  _runtime‚7020¹s2òí'¬"çO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117397859 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚278753683907072 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119030256 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12555017514496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚7020.167268724 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12772059258368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251317431203838 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117397859 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179969420 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚251029556291584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚569 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚263584386803200 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50499683 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚8899 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 297387317 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179969420 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117397857 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚730 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 297367277 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12554848514560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117397857 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚567 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚251317613364602 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12518828444026 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 % _timestamp‚1.7884023955301616e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚251029556291584 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚251029538288640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12771844479488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416ÅÃX‹`­"ìO _runtime‚7020.167268724 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚1277•—ã*’&2059258368 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚251317431203838 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 117397859 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 179969420 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚251029556291584 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚569 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚263584386803200 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚50499683 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚8899 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 297387317 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 179969420 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 117397857 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 _step‚730 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 297367277 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12554848514560 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 117397857 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚567 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚251317613364602 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12518828444026 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 % _timestamp‚1.7884023955301616e+09 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚251029556291584 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚251029538288640 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12771844479488 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 117397859 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚278753683907072 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119030256 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12555017514496 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160Ú Æ<2®"- _wandbruntime‚7020  _runtime‚7020ãšw÷&¯"ñM H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 297419789 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12556410665984 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297419789 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚251056825656320 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12556241652736 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119042431 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117410516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚251056807653376 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12520217608778 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚251056825656320 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68537383 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50505050 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179989207 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚263613236322304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚238824711063552 trainer/global_step‚8899 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 180009271 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚278784011124736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 180009271 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 179989207 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚263613049306112 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12556410665984 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50505567 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚251056807653376 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚278783780807680 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12773473527808 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚251344945711690 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚263613236322304 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚251344945711690 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚263613236322304 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 297419789 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚251056825656320 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12556241652736 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119042431 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 117410516 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚517 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚251056807653376 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12520217608778 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚519 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚251056825656320 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68537383 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50505050 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 179989207 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚263613236322304 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚238824711063552 trainer/global_step‚8899 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 180009271 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚278784011124736 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2Û•¶"2±"- _wandbruntime‚7020  _runtime‚7020º¢©ƒ]²"× /loss/FRt3/assignment_loss‚3.290854866440232 0loss/FRt4/assignment_loss‚1.2791498978182954 0loss/FBt2/assignment_loss‚0.6273257761163222 2loss/FBt3/assignment_loss‚0.009935269519435982 /loss/FRt2/detection_loss‚2.6434505845225136 %loss/total_loss‚43.95762024415413 % _timestamp‚1.7884023962023652e+09 2loss/SRqqt4/assignment_loss‚1.0362456028039257 1loss/SRqqt3/detection_loss‚1.7805751033273398 /loss/FBt2/detection_loss‚2.3016773787411777 trainer/global_step‚8899 2loss/SRqqt1/assignment_loss‚1.3147526160845984 1loss/FBt4/assignment_loss‚0.12335107189624193 /loss/FBt1/detection_loss‚2.7749074498812356 _step‚732 .loss/FRt2/assignment_loss‚4.44434563503709 0loss/SRqqt4/detection_loss‚1.319990830584651 /loss/FRt4/detection_loss‚1.8430983498692513 _runtime‚7020.839133978 2loss/SRqqt2/assignment_loss‚2.1647181325789653 .loss/FRt1/detection_loss‚1.993330498013672 1loss/SRqqt1/detection_loss‚1.7060646042227745 epoch‚7 0loss/FRt1/assignment_loss‚2.5088590990033066 1loss/FBt1/assignment_loss‚0.17998826126859058 .loss/FBt3/detection_loss‚3.325229952732722 2loss/SRqqt3/assignment_loss‚1.4870678282835905 /loss/FRt3/detection_loss‚2.6733646896239875 1loss/SRqqt2/detection_loss‚2.4069303508960838 /loss/FBt4/detection_loss‚0.7224063948881239æ‚KÜb³"Ü 2loss/SRqqt2/assignment_loss‚2.1647181325789653 .loss/FRt1/detection_loss‚1.993330498013672 1loss/SRqqt1/detection_loss‚1.7060646042227745 epoch‚7 0loss/FRt1/assignment_loss‚2.5088590990033066 1loss/FBt1/assignment_loss‚0.17998826126859058 .loss/FBt3/detection_loss‚3.325229952732722 2loss/SRqqt3/assignment_loss‚1.4870678282835905 /loss/FRt3/detection_loss‚2.6733646896239875 1loss/SRqqt2/detection_loss‚2.4069303508960838 /loss/FBt4/detection_loss‚0.7224063948881239 /loss/FRt3/assignment_loss‚3.290854866440232 0loss/FRt4/assignment_loss‚1.2791498978182954 0loss/FBt2/assignment_loss‚0.6273257761163222 2loss/FBt3/assignment_loss‚0.009935269519435982 /loss/FRt2/detection_loss‚2.6434505845225136 %loss/total_loss‚43.95762024415413 % _timestamp‚1.7884023962023652e+09 2loss/SRqqt4/assignment_loss‚1.0362456028039257 1loss/SRqqt3/detection_loss‚1.7805751033273398 /loss/FBt2/detection_loss‚2.3016773787411777 trainer/global_step‚8899 2loss/SRqqt1/assignment_loss‚1.3147526160845984 1loss/FBt4/assignment_loss‚0.12335107189624193 /loss/FBt1/detection_loss‚2.7749074498812356 _step‚732 .loss/FRt2/assignment_loss‚4.44434563503709 0loss/SRqqt4/detection_loss‚1.319990830584651 /loss/FRt4/detection_loss‚1.8430983498692513 _runtime‚7020.839133978Üåßà ´"‚(:’ êµãÔÀ¯¦® gpujÎÀŸ}.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚142.932' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.586399999999994 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ìc¯|µ"‚(:ñ êµãÔéøª° network.recv‚ 2.481871e+06 disk.rbd1.out‚ 17.29296875& disk.nvme0n1.out‚35172.08251953125 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.rbd0.out‚0* proc.memory.percent‚16.388988494873047) disk./.usagePercent‚15.67073881808244 disk.sda3.out‚ 8003.89453125 disk.sda.in‚ 0.06640625 disk.sda.out‚ 8003.89453125 disk.sda1.in‚0 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.nvme0n1.in‚ 399.828125 disk.sda2.out‚0 disk.dm-1.in‚ 0.06640625 disk.dm-1.out‚ 8003.89453125 disk.dm-0.in‚ 399.828125 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 399.828125 disk.drbd2265.in‚0 disk.rbd0.in‚0$ memory_percent‚96.00329399108887 network.sent‚ 2.1987718e+07 disk.drbd2080.out‚0! proc.memory.rssMB‚ 2685.171875% disk./.usageGB‚233.50533294677734( disk.nvme0c0n1.out‚35172.08251953125 disk.sda2.in‚0 cpu‚25.36420324455602 disk.sda3.in‚ 0.06640625 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0# disk.dm-0.out‚35172.08251953125' proc.memory.availableMB‚ 654.8203125ŸA)ú’¶"‚(:‡ ùµãÔ¼¦²® gpu.0.gpu‚33.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚122.154' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4308 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÀŠDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚265017526362624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚252393006525440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 299012969 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118030807 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚267427272059904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119670569 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119670022 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118030809 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12624519837184 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚8949 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 180962122 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚545 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚252684062139046 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚252393024528384 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118030809 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50771713 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚265017526362624 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12624519837184 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 180962122 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68898856 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0É•¨4ò'½"ìO JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚280270197299712 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚68898854 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12842925239808 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12588300262434 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12624688837120 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚280269960677888 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚240095926998016 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚252393006525440 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 180982160 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 180982160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚265017713365504 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12842712558080 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _runtime‚7055.881512915 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚265017526362624 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚252393006525440 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 299012969 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 118030807 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚267427272059904 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 119670569 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 119670022 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118030809 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12624519837184 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚8949 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 180962122 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚545 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚252684062139046 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚252393024528384 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 118030809 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚50771713 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚265017526362624 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12624519837184 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 180962122 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚68898856 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0Þ2eÒ2¾"- _wandbruntime‚7056  _runtime‚7056—É‘Ÿ!¿"ñM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119682827 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299045441 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚280300527187968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 181001973 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚252711394485926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚240121887219712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12589524305574 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚265046375881728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12844342179840 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 119683331 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 trainer/global_step‚8949 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚267456161068032 _step‚735 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚265046188865536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚502 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚240121870180352 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118043468 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118043466 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180981909 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12844133705728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884024319152517e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚252420293893120 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50776596 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12589689427186 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50777098 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 299025375 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚265046375881728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚252420293893120 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚252420275890176 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 180981909 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181001973 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚7056.552570946 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118043468 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68906233 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68906231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚267456185008128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 299045441 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12626081988608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚252420275890176 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118043466 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚252711576646898 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚504 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12625912975360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚280300294773760Ä£Àü&À"öM OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 181001973 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚7056.552570946 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 118043468 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚68906233 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚68906231 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚267456185008128 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 299045441 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12626081988608 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚252420275890176 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118043466 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚252711576646898 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚504 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12625912975360 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚280300294773760 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 119682827 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 299045441 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚280300527187968 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 181001973 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚252711394485926 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚240121887219712 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12589524305574 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚265046375881728 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12844342179840 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 119683331 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 trainer/global_step‚8949 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚267456161068032 _step‚735 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚265046188865536 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚502 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚240121870180352 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118043468 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 118043466 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 180981909 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12844133705728 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 % _timestamp‚1.7884024319152517e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚252420293893120 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚50776596 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12589689427186 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚50777098 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 299025375 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚265046375881728 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚252420293893120 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚252420275890176 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 180981909 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0߇52Á"- _wandbruntime‚7056  _runtime‚7056u¹øVÂ"Ð _step‚736 /loss/FRt2/assignment_loss‚3.498410209229118 2loss/SRqqt4/assignment_loss‚1.0328278726026896 0loss/FBt4/assignment_loss‚0.1913147414448255 /loss/FBt4/detection_loss‚0.5899508735386744 _runtime‚7056.553990794 /loss/FRt1/assignment_loss‚3.489748365170247 0loss/FRt3/assignment_loss‚2.9799980029463766 .loss/FRt4/detection_loss‚1.531321772130636 trainer/global_step‚8949 .loss/FRt2/detection_loss‚2.566248915697399 .loss/FRt3/detection_loss‚2.405360434576869 0loss/SRqqt2/detection_loss‚2.657750031043743 1loss/SRqqt3/detection_loss‚1.8771325170993804 %loss/total_loss‚43.98554830035074 /loss/FRt4/assignment_loss‚1.736366382928995 2loss/SRqqt1/assignment_loss‚1.2683316304215364 1loss/FBt2/assignment_loss‚0.34030129649346025 .loss/FBt2/detection_loss‚2.567372345007383 0loss/FBt3/assignment_loss‚0.2286734476685524 .loss/FRt1/detection_loss‚2.531786747276783 -loss/FBt3/detection_loss‚2.52006077170372 % _timestamp‚1.7884024319169505e+09 2loss/SRqqt3/assignment_loss‚1.4516950272023679 1loss/FBt1/assignment_loss‚0.03397036646424567 1loss/SRqqt4/detection_loss‚1.0306020735973311 /loss/FBt1/detection_loss‚3.9115694016218185 epoch‚7 2loss/SRqqt2/assignment_loss‚1.7450958819224918 1loss/SRqqt1/detection_loss‚1.7996591925621033«BÒP[Ã"Õ trainer/global_step‚8949 .loss/FRt2/detection_loss‚2.566248915697399 .loss/FRt3/detection_loss‚2.405360434576869 0loss/SRqqt2/detection_loss‚2.657750031043743 1loss/SRqqt3/detection_loss‚1.8771325170993804 %loss/total_loss‚43.98554830035074 /loss/FRt4/assignment_loss‚1.736366382928995 2loss/SRqqt1/assignment_loss‚1.2683316304215364 1loss/FBt2/assignment_loss‚0.34030129649346025 .loss/FBt2/detection_loss‚2.567372345007383 0loss/FBt3/assignment_loss‚0.2286734476685524 .loss/FRt1/detection_loss‚2.531786747276783 -loss/FBt3/detection_loss‚2.52006077170372 % _timestamp‚1.7884024319169505e+09 2loss/SRqqt3/assignment_loss‚1.4516950272023679 1loss/FBt1/assignment_loss‚0.03397036646424567 1loss/SRqqt4/detection_loss‚1.0306020735973311 /loss/FBt1/detection_loss‚3.9115694016218185 epoch‚7 2loss/SRqqt2/assignment_loss‚1.7450958819224918 1loss/SRqqt1/detection_loss‚1.7996591925621033 _step‚736 /loss/FRt2/assignment_loss‚3.498410209229118 2loss/SRqqt4/assignment_loss‚1.0328278726026896 0loss/FBt4/assignment_loss‚0.1913147414448255 /loss/FBt4/detection_loss‚0.5899508735386744 _runtime‚7056.553990794 /loss/FRt1/assignment_loss‚3.489748365170247 0loss/FRt3/assignment_loss‚2.9799980029463766 .loss/FRt4/detection_loss‚1.531321772130636à“q’Ä"‚(:‡ ˆ¶ãÔóË´® gpu.0.gpu‚36.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚136.314' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.2628 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ë-$áyÅ"‚(:î ˆ¶ãÔá‰Î°* proc.memory.percent‚16.388988494873047 network.sent‚ 2.2102366e+07 disk.drbd2266.out‚0 disk.drbd2080.out‚0# disk.dm-0.out‚35174.64404296875! proc.memory.rssMB‚ 2685.171875* disk./.usagePercent‚15.670803587988601 disk.dm-0.in‚ 399.828125 disk.nvme0c0n1.in‚ 399.828125 disk.sda.in‚ 0.06640625 proc.cpu.threads‚17 disk.rbd1.out‚ 17.4189453125 disk.sda.out‚ 8004.859375' proc.memory.availableMB‚ 654.6796875 cpu‚25.36287169009425 network.recv‚ 2.492616e+06% disk./.usageGB‚233.50629806518555$ memory_percent‚96.00415229797363( disk.nvme0c0n1.out‚35174.64404296875 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.06640625 disk.rbd1.in‚0 disk.dm-1.out‚ 8004.859375 disk.drbd2265.in‚0 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 8004.859375& disk.nvme0n1.out‚35174.64404296875 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 399.828125 disk.dm-1.in‚ 0.06640625p)²Æ"‚(:’ —¶ãÔŠ¡ ® gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚136.974' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.394799999999996 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0£²ÆoÇ"‚(:ä —¶ãÔßà€± disk.sda3.out‚8005.375 disk.rbd1.out‚ 17.4814453125 disk.sda.in‚ 0.06640625 disk.drbd1716.in‚0 disk.rbd0.out‚0# memory_percent‚96.0033655166626* proc.memory.percent‚16.388988494873047 disk.nvme0c0n1.in‚ 399.828125 disk.drbd2266.out‚0 disk.dm-0.in‚ 399.828125 network.recv‚ 2.499251e+06 disk.drbd2265.out‚0 disk.drbd2080.in‚0# disk.dm-0.out‚35174.92919921875 disk.sda3.in‚ 0.06640625 disk.drbd2080.out‚0 cpu‚25.36220818885192 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.06640625 network.sent‚ 2.221121e+07* disk./.usagePercent‚15.670776963205041 disk.sda2.in‚0 disk.dm-1.out‚8005.375( proc.memory.availableMB‚ 654.80859375 disk.rbd1.in‚0( disk.nvme0c0n1.out‚35174.92919921875& disk.nvme0n1.out‚35174.92919921875 disk.sda.out‚8005.375 disk.sda2.out‚0 disk.drbd2266.in‚0% disk./.usageGB‚233.50590133666992 disk.sda1.in‚0 disk.nvme0n1.in‚ 399.828125 disk.drbd2265.in‚0 disk.rbd0.in‚0! proc.memory.rssMB‚ 2685.171875 disk.sda1.out‚0ç´Þ2È"- _wandbruntime‚7091  _runtime‚7091phøÛÇÉ"Á _runtime‚7091.396682093 _step‚737 & lr-AdamW/pg1‚4.003992001338782e-05 & lr-AdamW/pg2‚4.003992001338782e-05 trainer/global_step‚8999 % _timestamp‚1.7884024667598548e+09®.¶ýÌÊ"Æ & lr-AdamW/pg1‚4.003992001338782e-05 & lr-AdamW/pg2‚4.003992001338782e-05 trainer/global_step‚8999 % _timestamp‚1.7884024667598548e+09 _runtime‚7091.396682093 _step‚737á“Ô2Ë"- _wandbruntime‚7091  _runtime‚7091‡¦‘ì'Ì"æO D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚241393086114816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120310174 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118663759 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚253756492765184 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12694191159808 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚253756492765184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 300638621 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181954824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚253756474762240 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12694360159744 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚268872919471104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69267706 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚571 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51043041 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚241393103154176 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 $ _timestamp‚1.788402466760548e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12694360159744 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚266450665922048 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12694191159808 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120310747 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0Üm…0fÍ"ëO KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51042470 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 181954824 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚12913901159424 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚254050693074254 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 181974862 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _runtime‚7091.397570719 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚253756474762240 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚241393086114816 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120310174 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 118663759 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚253756492765184 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12694191159808 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚253756492765184 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 300638621 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 181954824 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚253756474762240 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12694360159744 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚268872919471104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69267706 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 QIDeviceStatsMo³Oìì‹nitor.on_train_batch_start/inactive_split.small_pool.current‚571 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51043041 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚241393103154176 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 $ _timestamp‚1.788402466760548e+09 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12694360159744 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚266450665922048 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12694191159808 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120310747 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚281816920496640 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120323004 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300671093 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120323521 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚253783762129920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚254078207582106 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚266479328424960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚266479515441152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚241419046336512 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚7092.066314608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 181974611 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118676418 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚253783762129920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300651027 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69275083 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚241419029297152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12695584297984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300651027 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚254078025421134 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300671093 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚739 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12659161245594 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51048438 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12915112017408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚253783744126976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181974611 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118676416 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12695753311232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚268901832419328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚266479515441152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217Ïú†Œü&Ð"öM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚241419029297152 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12695584297984 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 300651027 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚254078025421134 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 300671093 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 _step‚739 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12659161245594 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51048438 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12915112017408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚253783744126976 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 181974611 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 118676416 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12695753311232 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚268901832419328 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚266479515441152 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚281816920496640 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120323004 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 300671093 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120323521 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚253783762129920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚254078207582106 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚266479328424960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚266479515441152 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚241419046336512 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚7092.066314608 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 181974611 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 118676418 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚253783762129920 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 300651027 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69275083 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0ãç6´^2Ñ"- _wandbruntime‚7092  _runtime‚7092gEËZÒ"Ô /loss/FBt1/detection_loss‚2.5373668521642685 0loss/FRt4/assignment_loss‚1.6015403485298156 0loss/FBt2/assignment_loss‚1.0110901196796174 1loss/SRqqt3/detection_loss‚1.9782829575088559 epoch‚7 trainer/global_step‚8999 2loss/SRqqt3/assignment_loss‚1.5204385779037768 0loss/FBt3/assignment_loss‚0.3977591932634823 _step‚740 /loss/FRt4/detection_loss‚1.9542003513127566 .loss/FBt3/detection_loss‚2.348546653985977 %loss/total_loss‚47.22893162427839 /loss/FRt3/assignment_loss‚3.091987006592028 2loss/SRqqt1/assignment_loss‚1.5975544836757511 /loss/FRt3/detection_loss‚2.6321889026598497 .loss/FBt2/detection_loss‚5.615527232487996 _runtime‚7092.067284869 1loss/FBt4/assignment_loss‚0.13606627785012862 /loss/FRt1/detection_loss‚1.8446238347978303 /loss/FRt2/detection_loss‚2.8521529295865227 1loss/SRqqt1/detection_loss‚1.8107327626183116 /loss/FBt4/detection_loss‚0.6546426210785284 /loss/FRt1/assignment_loss‚3.438441121216976 /loss/FRt2/assignment_loss‚3.305064394193537 2loss/SRqqt2/assignment_loss‚1.8512585335544178 0loss/SRqqt2/detection_loss‚2.372379129273551 % _timestamp‚1.7884024674305375e+09 2loss/SRqqt4/assignment_loss‚0.8408714668176165 0loss/FBt1/assignment_loss‚0.7603568106169405 1loss/SRqqt4/detection_loss‚1.0758590629098541EŽE­_Ó"Ù 0loss/FRt4/assignment_loss‚1.6015403485298156 0loss/FBt2/assignment_loss‚1.0110901196796174 1loss/SRqqt3/detection_loss‚1.9782829575088559 epoch‚7 trainer/global_step‚8999 2loss/SRqqt3/assignment_loss‚1.5204385779037768 0loss/FBt3/assignment_loss‚0.3977591932634823 _step‚740 /loss/FRt4/detection_loss‚1.9542003513127566 .loss/FBt3/detection_loss‚2.348546653985977 %loss/total_loss‚47.22893162427839 /loss/FRt3/assignment_loss‚3.091987006592028 2loss/SRqqt1/assignment_loss‚1.5975544836757511 /loss/FRt3/detection_loss‚2.6321889026598497 .loss/FBt2/detection_loss‚5.615527232487996 _runtime‚7092.067284869 1loss/FBt4/assignment_loss‚0.13606627785012862 /loss/FRt1/detection_loss‚1.8446238347978303 /loss/FRt2/detection_loss‚2.8521529295865227 1loss/SRqqt1/detection_loss‚1.8107327626183116 /loss/FBt4/detection_loss‚0.6546426210785284 /loss/FRt1/assignment_loss‚3.438441121216976 /loss/FRt2/assignment_loss‚3.305064394193537 2loss/SRqqt2/assignment_loss‚1.8512585335544178 0loss/SRqqt2/detection_loss‚2.372379129273551 % _timestamp‚1.7884024674305375e+09 2loss/SRqqt4/assignment_loss‚0.8408714668176165 0loss/FBt1/assignment_loss‚0.7603568106169405 1loss/SRqqt4/detection_loss‚1.0758590629098541 /loss/FBt1/detection_loss‚2.5373668521642685äÄÒý5’Ô"‚(:‡ ¦¶ãÔä¾ã® gpu.0.gpu‚33.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚136.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.3466 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ãMÙvÕ"‚(:ë ¦¶ãÔàçý¯$ disk.nvme0n1.out‚35175.541015625 disk.sda1.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 399.828125& disk.nvme0c0n1.out‚35175.541015625 disk.rbd0.in‚0 network.sent‚ 2.2216605e+07 disk.nvme0n1.in‚ 399.828125 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0! disk.dm-0.out‚35175.541015625 disk.sda3.out‚ 8006.0234375 disk.sda.in‚ 0.06640625 disk.sda.out‚ 8006.0234375 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.nvme0c0n1.in‚ 399.828125* proc.memory.percent‚16.388988494873047 network.recv‚ 2.502711e+06 disk.dm-1.out‚ 8006.0234375 disk.sda1.in‚0 disk.rbd1.in‚0' proc.memory.availableMB‚ 654.3515625 cpu‚25.36155807878849 disk.rbd1.out‚ 17.4814453125$ memory_percent‚96.00615501403809 proc.cpu.threads‚17* disk./.usagePercent‚15.670775427159835 disk.sda2.out‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2265.in‚0 disk.sda3.in‚ 0.06640625 disk.sda2.in‚0! proc.memory.rssMB‚ 2685.171875% disk./.usageGB‚233.50587844848633/’Ö"‚(:‡ µ¶ãÔè’˜® gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚141.817' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.3634 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é| Ýk×"‚(:à µ¶ãÔå˲° disk.rbd0.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.06640625$ disk.nvme0n1.out‚35175.966796875( proc.memory.availableMB‚ 654.55859375 network.recv‚ 2.508749e+06% disk./.usageGB‚233.50636672973633 disk.drbd2266.in‚0 disk.rbd0.in‚0 disk.sda3.out‚ 8006.5625 network.sent‚ 2.232508e+07 disk.drbd1716.in‚0 disk.drbd2265.out‚0! disk.dm-0.out‚35175.966796875 proc.cpu.threads‚17* disk./.usagePercent‚15.670808196124216$ memory_percent‚96.00489139556885 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0& disk.nvme0c0n1.out‚35175.966796875* proc.memory.percent‚16.388988494873047 disk.drbd1716.out‚0 disk.dm-0.in‚ 399.828125 disk.rbd1.out‚ 17.693359375 disk.nvme0c0n1.in‚ 399.828125 disk.nvme0n1.in‚ 399.828125 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.06640625 disk.dm-1.out‚ 8006.5625 disk.sda1.in‚0 disk.rbd1.in‚0! proc.memory.rssMB‚ 2685.171875 cpu‚25.3609000102712 disk.sda.in‚ 0.06640625 disk.sda.out‚ 8006.5625 disk.drbd2266.out‚0´PdjØ"‚(:’ ĶãÔ·¾ƒ® gpu.0.gpu‚17.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚111.353' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.270599999999998 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0™ãÕrhÙ"‚(:î ĶãÔØÆô¯ disk.drbd2265.in‚0( proc.memory.availableMB‚ 654.80078125 cpu‚25.36025137907555 proc.cpu.threads‚17 disk.sda.in‚ 0.06640625 disk.drbd2265.out‚0# disk.dm-0.out‚35177.71142578125 disk.rbd1.out‚ 17.693359375( disk.nvme0c0n1.out‚35177.71142578125* proc.memory.percent‚16.388988494873047& disk.nvme0n1.out‚35177.71142578125 disk.sda2.out‚0 disk.rbd0.in‚0 disk.rbd0.out‚0 disk.sda3.out‚ 8007.109375! proc.memory.rssMB‚ 2685.171875* disk./.usagePercent‚15.670807428101616 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.sda1.out‚0 network.sent‚ 2.2331142e+07 disk.dm-1.in‚ 0.06640625 disk.nvme0c0n1.in‚ 399.828125% disk./.usageGB‚233.50635528564453 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 399.828125(…}3 disk.rbd1.in‚0 network.recv‚ 2.513018e+06 disk.nvme0n1.in‚ 399.828125 disk.drbd2080.in‚0 disk.sda3.in‚ 0.06640625$ memory_percent‚96.00341320037842 disk.sda2.in‚0 disk.dm-1.out‚ 8007.109375 disk.sda1.in‚0 disk.sda.out‚ 8007.109375´TØE2Ú"- _wandbruntime‚7127  _runtime‚7127ŤóÐÇÛ"Á & lr-AdamW/pg1‚3.883875800682895e-05 & lr-AdamW/pg2‚3.883875800682895e-05 trainer/global_step‚9049 % _timestamp‚1.7884025024130323e+09 _runtime‚7127.049880118 _step‚7416ÀÂÌÜ"Æ & lr-AdamW/pg1‚3.883875800682895e-05 & lr-AdamW/pg2‚3.883875800682895e-05 trainer/global_step‚9049 % _timestamp‚1.7884025024130323e+09 _runtime‚7127.049880118 _step‚741åO•²_2Ý"- _wandbruntime‚7127  _runtime‚71272še÷í'Þ"çO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51314101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69636554 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12763862482432 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚255417324009462 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚255119942999040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12984605438976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119296707 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 182947526 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120950657 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚255119942999040 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12764031482368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 302264273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51313564 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267883805481472 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚270318566882304 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 _runtime‚7127.050827745 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69636556 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267883992484352 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12764031482368 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 182967564 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚270318542942208 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119296707 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12727078777846 _step‚742 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267883992484352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0[Uh:ò'ß"ìO C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12763862482432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚283303382905856 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚255119961001984 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 182967564 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚242690262270976 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119296709 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12727243899250 % _timestamp‚1.7884025024137585e+09 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 120950118 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚255119942999040 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12764031482368 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 302264273 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51313564 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚267883805481472 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚270318566882304 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 _runtime‚7127.050827745 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚69636556 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚267883992484352 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12764031482368 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 182967564 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚270318542942208 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119296707 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12727078777846 _step‚742 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚539 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚267883992484352 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51314101 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚69636554 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12763862482432 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚255417324009462 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚255119942999040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚12984605438976 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119296707 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 182947526 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 120950657 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119309368 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 302276679 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12765424633856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12765255620608 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302296745 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120962838 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119309366 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69643933 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69643931 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 trainer/global_step‚9049 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 302276679 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚255444838517314 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120963370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12986028564992 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚283333714772480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚270347479830528 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚12986234941952 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚255147212363776 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119309366 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _runtime‚7127.725527379 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12728633064002 _step‚743 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 182987377 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚242716188413952 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚255147212363776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚255147212363776 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119309366 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _runtime‚7127.725527379 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12728633064002 _step‚743 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 182987377 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.f’¬ÈïP!reed‚242716188413952 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚255147212363776 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 119309368 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 302276679 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12765424633856 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12765255620608 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 302296745 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 120962838 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119309366 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚69643933 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚69643931 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 trainer/global_step‚9049 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 302276679 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚255444838517314 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 120963370 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚12986028564992 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚283333714772480 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0çtÖ2ã"- _wandbruntime‚7127  _runtime‚7127>Ñž¼Uä"Ï 0loss/FRt4/assignment_loss‚1.5230456814169884 2loss/SRqqt2/assignment_loss‚2.9226954486221075 1loss/FBt2/assignment_loss‚0.22939034418664997 0loss/SRqqt3/detection_loss‚1.727835458123459 /loss/FBt2/detection_loss‚2.9410759756962457 .loss/FBt3/detection_loss‚2.857649736934238 /loss/FRt2/assignment_loss‚3.872081813812256 .loss/FRt3/detection_loss‚2.717204161709355 % _timestamp‚1.7884025030901017e+09 _step‚744 /loss/FRt1/assignment_loss‚2.787848244492824 2loss/SRqqt1/assignment_loss‚1.5083871199039078 0loss/FBt1/assignment_loss‚0.4565825170470362 /loss/FBt1/detection_loss‚2.3598751187324525 .loss/FRt1/detection_loss‚2.061512371668449 -loss/FRt2/detection_loss‚3.11156756401062 .loss/FRt4/detection_loss‚2.200278057903051 1loss/SRqqt4/detection_loss‚1.2383675611345097 /loss/FBt4/detection_loss‚0.5862451441271967 trainer/global_step‚9049 _runtime‚7127.726872467 2loss/SRqqt3/assignment_loss‚1.5496339705192819 1loss/FBt4/assignment_loss‚0.16881587630678305 /loss/FRt3/assignment_loss‚2.184482443717218 /loss/FBt3/assignment_loss‚0.250633935754396 0loss/SRqqt2/detection_loss‚2.342180633917451 2loss/SRqqt4/assignment_loss‚0.9769378795172088 1loss/SRqqt1/detection_loss‚1.5735390835394294 %loss/total_loss‚44.14786614279311 epoch‚7t?Zå"Ô .loss/FBt3/detection_loss‚2.857649736934238 /loss/FRt2/assignment_loss‚3.872081813812256 .loss/FRt3/detection_loss‚2.717204161709355 % _timestamp‚1.7884025030901017e+09 _step‚744 /loss/FRt1/assignment_loss‚2.787848244492824 2loss/SRqqt1/assignment_loss‚1.5083871199039078 0loss/FBt1/assignment_loss‚0.4565825170470362 /loss/FBt1/detection_loss‚2.3598751187324525 .loss/FRt1/detection_loss‚2.061512371668449 -loss/FRt2/detection_loss‚3.11156756401062 .loss/FRt4/detection_loss‚2.200278057903051 1loss/SRqqt4/detection_loss‚1.2383675611345097 /loss/FBt4/detection_loss‚0.5862451441271967 trainer/global_step‚9049 _runtime‚7127.726872467 2loss/SRqqt3/assignment_loss‚1.5496339705192819 1loss/FBt4/assignment_loss‚0.16881587630678305 /loss/FRt3/assignment_loss‚2.184482443717218 /loss/FBt3/assignment_loss‚0.250633935754396 0loss/SRqqt2/detection_loss‚2.342180633917451 2loss/SRqqt4/assignment_loss‚0.9769378795172088 1loss/SRqqt1/detection_loss‚1.5735390835394294 %loss/total_loss‚44.14786614279311 epoch‚7 0loss/FRt4/assignment_loss‚1.5230456814169884 2loss/SRqqt2/assignment_loss‚2.9226954486221075 1loss/FBt2/assignment_loss‚0.22939034418664997 0loss/SRqqt3/detection_loss‚1.727835458123459 /loss/FBt2/detection_loss‚2.9410759756962457èwJ˜\æ"‚(:… Ó¶ãÔÊ˧® gpu.0.gpu‚28.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚122.98' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.596 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P!Ò2|ç"‚(:ñ Ó¶ãÔ€ž² disk.rbd1.out‚ 17.755859375 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 399.828125 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.sda.in‚ 0.06640625 disk.dm-1.in‚ 0.06640625 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.dm-0.in‚ 399.828125# disk.dm-0.out‚35177.99267578125' proc.memory.availableMB‚ 654.4921875! proc.memory.rssMB‚ 2685.171875 cpu‚25.359584773401554 network.recv‚ 2.517491e+06* disk./.usagePercent‚15.670804868026272 disk.dm-1.out‚ 8007.4609375 disk.drbd2265.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.06640625( disk.nvme0c0n1.out‚35177.99267578125& disk.nvme0n1.out‚35177.99267578125 disk.rbd0.out‚0$ memory_percent‚96.00529670715332 disk.sda1.in‚0% disk./.usageGB‚233.50631713867188 disk.nvme0n1.in‚ 399.828125 disk.sda.out‚ 8007.4609375 disk.rbd0.in‚0 disk.sda3.out‚ 8007.4609375* proc.memory.percent‚16.388988494873047 proc.cpu.threads‚17 network.sent‚ 2.2337476e+07 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0ƒXP‡è"‚(:… â¶ãÔ¼°® gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚107.99' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.598 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0šíh~é"‚(:ó â¶ãÔ‚¦ž° disk.dm-1.out‚ 8008.22265625 disk.drbd2080.out‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚35178.25439453125$ memory_percent‚96.00322246551514* proc.memory.percent‚16.388988494873047 network.sent‚ 2.244554e+07 disk.sda.in‚ 0.06640625 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.dm-0.in‚ 399.828125 disk.sda1.out‚0 network.recv‚ 2.522893e+06& disk.nvme0n1.out‚35178.25439453125 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 399.828125 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0# disk.dm-0.out‚35178.25439453125 disk.sda3.out‚ 8008.22265625 disk.rbd1.out‚ 17.755859375* disk./.usagePercent‚15.670801027913258$ disk./.usageGB‚233.5062599182129 disk.dm-1.in‚ 0.06640625 disk.drbd2266.out‚0 disk.sda3.in‚ 0.06640625 disk.nvme0c0n1.in‚ 399.828125 disk.drbd2080.in‚0 disk.sda.out‚ 8008.22265625 disk.sda1.in‚0( proc.memory.availableMB‚ 654.83203125 proc.cpu.threads‚17! proc.memory.rssMB‚ 2685.171875 cpu‚25.358952621623114]Ó2ê"- _wandbruntime‚7162  _runtime‚7162:ΗÇë"Á & lr-AdamW/pg2‚3.765190350534087e-05 trainer/global_step‚9099 % _timestamp‚1.7884025379265656e+09 _runtime‚7162.563431946 _step‚745 & lr-AdamW/pg1‚3.765190350534087e-058P Ìì"Æ & lr-AdamW/pg2‚3.765190350534087e-05 trainer/global_step‚9099 % _timestamp‚1.7884025379265656e+09 _runtime‚7162.563431946 _step‚745 & lr-AdamW/pg1‚3.765190350534087e-05é¦v2í"- _wandbruntime‚7162  _runtime‚7162·l¸Sí'î"çO ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚256483429238784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚9099 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 303889925 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51585450 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚269316945040896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚269317132043776 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12796550596254 % _timestamp‚1.7884025379272938e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119929657 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119929659 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚256483411235840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚509 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚256483429238784 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121590345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70005404 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚256783954944670 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119929657 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12833533805056 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 121590856 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416)ÿ–²oï"ìO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 119929659 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚256483411235840 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚509 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚256483429238784 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 121590345 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70005404 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚256783954944670 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 119929657 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12833533805056 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 121590856 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚256483429238784 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚9099 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 303889925 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚51585450 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚269316945040896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚269317132043776 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12796550596254 % _timestamp‚1.7884025379272938e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 119929657 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0ê‰1åô2ð"- _wandbruntime‚7163  _runtime‚7163=µ×Åö&ñ"ðM L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 303922397 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788402538593462e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12834926943232 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13057140302336 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12798104882410 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚244013364570112 _runtime‚7163.230926809 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚256510698603520 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51590742 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚244013347530752 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 303902331 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12834926943232 trainer/global_step‚9099 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121602993 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12835095956480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12797939760798 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚256510698603520 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚256510680600576 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚269345794560000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _step‚747 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚269345607543808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51590212 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚284850267544064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚256811287291550 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚269345794560000 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚256510680600576 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70012783 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183960015 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚256811469452522 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119942316 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚284850024644096 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚271793127241728 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70012781 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121603525 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183980079 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119942316 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚271793103301632 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183980079 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚530ÑzAû&ò"õM cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚271793127241728 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70012781 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 121603525 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 183980079 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 119942316 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚271793103301632 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 183980079 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚530 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 303922397 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788402538593462e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12834926943232 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13057140302336 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12798104882410 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚244013364570112 _runtime‚7163.230926809 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚256510698603520 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51590742 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚244013347530752 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 303902331 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12834926943232 trainer/global_step‚9099 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 121602993 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚12835095956480 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12797939760798 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚256510698603520 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚256510680600576 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚532 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚269345794560000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _step‚747 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚269345607543808 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51590212 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚284850267544064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚256811287291550 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚269345794560000 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚256510680600576 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70012783 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 183960015 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚256811469452522 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 119942316 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚284850024644096ëýF|¾2ó"- _wandbruntime‚7163  _runtime‚7163c±!Ã\ô"Ö 1loss/SRqqt3/detection_loss‚1.8210362462947767 _runtime‚7163.232375041 0loss/FRt4/assignment_loss‚1.4078429735132627 2loss/SRqqt4/assignment_loss‚1.1105113165208795 1loss/FBt4/assignment_loss‚0.15868874033255054 /loss/FRt1/detection_loss‚2.2538011595606804 /loss/FRt2/detection_loss‚3.0348742665915656 /loss/SRqqt1/detection_loss‚2.00197248888964 %loss/total_loss‚44.55678807527697 0loss/FRt2/assignment_loss‚3.2037215643915635 /loss/FRt3/assignment_loss‚2.805200545489788 1loss/SRqqt4/detection_loss‚1.2338249420166993 /loss/FRt4/detection_loss‚1.8377815292749022 /loss/FBt1/detection_loss‚2.9151062284197127 2loss/SRqqt1/assignment_loss‚1.2062038052403792 .loss/FBt3/detection_loss‚3.869868357976278 0loss/FRt1/assignment_loss‚3.3709799349308014 1loss/SRqqt3/assignment_loss‚1.490651110264783 1loss/FBt3/assignment_loss‚0.06521292154987653 /loss/FRt3/detection_loss‚2.3503937616944315 epoch‚7 0loss/FBt1/assignment_loss‚0.5680924397934827 0loss/FBt2/assignment_loss‚0.4009474366053162 /loss/FBt2/detection_loss‚2.3162212091333725 _step‚748 2loss/SRqqt2/assignment_loss‚1.9986234727230938 1loss/SRqqt2/detection_loss‚2.5583393474419913 /loss/FBt4/detection_loss‚0.5768922766271446 trainer/global_step‚9099 % _timestamp‚1.7884025385955899e+09-çºaõ"Û 1loss/SRqqt3/assignment_loss‚1.490651110264783 1loss/FBt3/assignment_loss‚0.06521292154987653 /loss/FRt3/detection_loss‚2.3503937616944315 epoch‚7 0loss/FBt1/assignment_loss‚0.5680924397934827 0loss/FBt2/assignment_loss‚0.4009474366053162 /loss/FBt2/detection_loss‚2.3162212091333725 _step‚748 2loss/SRqqt2/assignment_loss‚1.9986234727230938 1loss/SRqqt2/detection_loss‚2.5583393474419913 /loss/FBt4/detection_loss‚0.5768922766271446 trainer/global_step‚9099 % _timestamp‚1.7884025385955899e+09 1loss/SRqqt3/detection_loss‚1.8210362462947767 _runtime‚7163.232375041 0loss/FRt4/assignment_loss‚1.4078429735132627 2loss/SRqqt4/assignment_loss‚1.1105113165208795 1loss/FBt4/assignment_loss‚0.15868874033255054 /loss/FRt1/detection_loss‚2.2538011595606804 /loss/FRt2/detection_loss‚3.0348742665915656 /loss/SRqqt1/detection_loss‚2.00197248888964 %loss/total_loss‚44.55678807527697 0loss/FRt2/assignment_loss‚3.2037215643915635 /loss/FRt3/assignment_loss‚2.805200545489788 1loss/SRqqt4/detection_loss‚1.2338249420166993 /loss/FRt4/detection_loss‚1.8377815292749022 /loss/FBt1/detection_loss‚2.9151062284197127 2loss/SRqqt1/assignment_loss‚1.2062038052403792 .loss/FBt3/detection_loss‚3.869868357976278 0loss/FRt1/assignment_loss‚3.3709799349308014ì uëö"‚(:… ñ¶ãԘ gpu.0.gpu‚22.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚146.42' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.284 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Sce÷"‚(:ö ñ¶ãÔÌËä± network.sent‚ 2.2451617e+07& disk.nvme0n1.out‚35178.75830078125 disk.drbd2266.out‚0 disk.rbd0.out‚0! proc.memory.rssMB‚ 2685.171875 disk.drbd2080.out‚0% disk./.usageGB‚233.50624084472656 disk.dm-0.in‚ 399.828125# disk.dm-0.out‚35178.75830078125 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.sda3.in‚ 0.06640625( disk.nvme0c0n1.out‚35178.75830078125 disk.drbd1716.out‚0* proc.memory.percent‚16.388988494873047 disk.rbd1.in‚0 disk.nvme0n1.in‚ 399.828125 disk.sda.out‚ 8008.65234375 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 8008.65234375 disk.nvme0c0n1.in‚ 399.828125 disk.dm-1.in‚ 0.06640625 disk.rbd0.in‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0( proc.memory.availableMB‚ 653.96484375 cpu‚25.358297207381295 network.recv‚ 2.527336e+06* disk./.usagePercent‚15.670799747875588 disk.rbd1.out‚ 17.8818359375 disk.sda.in‚ 0.06640625 disk.sda2.out‚0 disk.dm-1.out‚ 8008.65234375$ memory_percent‚96.00851535797119½g¦9ø"‚(:’ €·ãÔؽ†® gpu.0.gpu‚28.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚141.047' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.209400000000002 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0fnzù"‚(:ï €·ãÔÊ–‚±$ memory_percent‚96.00794315338135* proc.memory.percent‚16.388988494873047 disk.rbd0.in‚0 disk.rbd1.out‚ 17.8818359375 cpu‚25.357661469889546 disk.drbd2265.out‚0 network.recv‚ 2.539983e+06$ disk./.usageGB‚233.5063819885254 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.drbd1716.in‚0 network.sent‚ 2.2563822e+07* disk./.usagePercent‚15.670809220154355 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.sda.in‚ 0.06640625 disk.sda2.out‚0( proc.memory.availableMB‚ 654.05859375# disk.dm-0.out‚35179.02783203125 disk.sda3.in‚ 0.06640625 disk.sda3.out‚ 8009.484375 disk.dm-0.in‚ 399.828125 disk.sda1.out‚0 disk.rbd1.in‚0! proc.memory.rssMB‚ 2685.171875 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0( disk.nvme0c0n1.out‚35179.02783203125 disk.nvme0n1.in‚ 399.828125& disk.nvme0n1.out‚35179.02783203125 disk.drbd1716.out‚0 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 399.828125 disk.sda.out‚ 8009.484375 disk.dm-1.in‚ 0.06640625 disk.dm-1.out‚ 8009.484375áÕîþ2ú"- _wandbruntime‚7198  _runtime‚7198^ÞÀÆû"À & lr-AdamW/pg2‚3.647960384762283e-05 trainer/global_step‚9149 % _timestamp‚1.7884025737296286e+09 _runtime‚ 7198.36649584 _step‚749 & lr-AdamW/pg1‚3.647960384762283e-05ûíVöËü"Å trainer/global_step‚9149 % _timestamp‚1.7884025737296286e+09 _runtime‚ 7198.36649584 _step‚749 & lr-AdamW/pg1‚3.647960384762283e-05 & lr-AdamW/pg2‚3.647960384762283e-05í„ä2ý"- _wandbruntime‚7198  _runtime‚7198ÅÙ›Þ þ"åO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 $ _timestamp‚1.788402573730356e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12903205127680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚258150585879878 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 305515577 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13126434958336 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚286336272722944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚245284563465216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚270750271603200 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚286336507247616 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚270750084600320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12903205127680 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12866187536066 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 &ßð'ÿ"êO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚257846897475584 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 184952968 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _runtime‚ 7198.36736009 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 184932930 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚270750084600320 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _step‚750 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚51853766 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚270750084600320 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12903205127680 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12866187536066 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 $ _timestamp‚1.788402573730356e+09 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12903205127680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚258150585879878 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 305515577 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13126434958336 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚286336272722944 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚245284563465216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚270750271603200 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚286336507247616 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120575266 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12867576700818 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13127849667072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚258178100387730 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _step‚751 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚257874148837376 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122240706 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51859594 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚258177918226758 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 184952717 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚257874166840320 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚9149 % _timestamp‚1.7884025744060428e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 184972781 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚286366600379904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12904598265856 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 _runtime‚7199.043425348 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚521 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 184952717 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚270778934119424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70381631 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184972781 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51859075 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚286366836988416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 305527983 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305548049 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70381633 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚273238774652928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122241227 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12867411579206 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚257874166840320 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚9149 % _timestamp‚1.7884025744060428e+09 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 184972781 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚286366600379904 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12904598265856 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 _runtime‚7199.043425348 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚521 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 184952717 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚270778934119424 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70381631 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 184972781 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚51859075 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚286366836988416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 305527983 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 305548049 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚7038163«gèšE3 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚273238774652928 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 122241227 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12867411579206 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 120575266 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚12867576700818 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13127849667072 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚258178100387730 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _step‚751 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚257874148837376 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122240706 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚51859594 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚258177918226758 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 184952717 FDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70743106 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13197299625984 _step‚754 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12973045450240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚246581739621376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚287852784801792 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185925632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚259210365712384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52125154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚259517216815086 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚272183411162624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52125680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12972876450304 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12935494233070 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307121189 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 122868258 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚274655485175808 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚246581722582016 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121195557 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚259210347709440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚259210347709440 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185945670 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272183224159744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307141229 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 185925632 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199ó•kŒñ'‘#ëO [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚12972876450304 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚12935494233070 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 307121189 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 122868258 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚274655485175808 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚246581722582016 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121195557 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚259210347709440 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚259210347709440 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 185945670 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚272183224159744 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 307141229 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 185925632 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 trainer/global_step‚9199 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚12973045450240 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚12935659354474 % _timestamp‚1.7884026094257715e+09 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚70743104 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121195559 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚272183224159744 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 185945670 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚12972876450304 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚70743106 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13197299625984 _step‚754 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚12973045450240 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚246581739621376 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚287852784801792 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 185925632 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚259210365712384 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52125154 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚259517216815086 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚272183411162624 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52125680 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217ò÷½]2’#- _wandbruntime‚7234  _runtime‚7234 ÈÅ“#Šžù ó&ñM UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚70750483 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13198727624192 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚272211886662656 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52131038 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚12974269588480 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 185945419 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚287883125748224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13198934001152 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307173701 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 185965483 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚274684422064128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122881009 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚259544549161966 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12974269588480 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121208216 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚246607665764352 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚259237635077120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚512 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚272212073678848 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70750481 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121208216 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚259237617074176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12936883397614 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52130528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 % _timestamp‚1.7884026103236473e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12974438601728 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚274684398124032 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚259544731322938 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 307173701 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚9199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 185945419 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚287883125748224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13198934001152 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 307173701 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 185965483 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚274684422064128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 122881009 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚259544549161966 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚12974269588480 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121208216 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚246607665764352 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚259237635077120 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚512 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚272212073678848 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚70750481 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 121208216 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚259237617074176 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚12936883397614 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52130528 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 % _timestamp‚1.7884026103236473e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚12974438601728 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚274684398124032 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚259544731322938 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 307173701 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚9199 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0ó Ò×G2•#- _wandbruntime‚7234  _runtime‚72346VY–#Ó 2loss/SRqqt3/assignment_loss‚1.3725103992605354 1loss/FBt2/assignment_loss‚0.22202992531741472 1loss/SRqqt3/detection_loss‚2.1873213216298963 trainer/global_step‚9199 0loss/FRt1/assignment_loss‚3.1543760310519824 /loss/FRt2/assignment_loss‚3.933777098854383 .loss/FRt1/detection_loss‚1.988779091835039 0loss/SRqqt2/detection_loss‚2.304432614283128 _runtime‚7234.962362736 0loss/SRqqt4/detection_loss‚1.514065068276202 /loss/FBt2/detection_loss‚3.1674254462122917 2loss/SRqqt1/assignment_loss‚1.4646316517499232 1loss/FBt3/assignment_loss‚0.24031831696628386 -loss/FRt2/detection_loss‚3.18828180556496 _step‚756 2loss/SRqqt2/assignment_loss‚2.1105644007523856 2loss/SRqqt4/assignment_loss‚1.0042463907910908 /loss/FRt4/detection_loss‚2.0952839748087255 /loss/FBt3/detection_loss‚2.9052107453346254 /loss/FRt3/assignment_loss‚2.433178201664326 0loss/FRt4/assignment_loss‚1.8306168968027288 0loss/SRqqt1/detection_loss‚2.192033895424434 /loss/FBt1/detection_loss‚4.1714844306310015 &loss/total_loss‚47.463361128093496 0loss/FBt1/assignment_loss‚0.7739133449892203 1loss/FBt4/assignment_loss‚0.14819937090837723 /loss/FBt4/detection_loss‚0.6178721718538505 epoch‚7 -loss/FRt3/detection_loss‚2.44280853313069 % _timestamp‚1.7884026103256147e+09¢.é€^—#Ø _runtime‚7234.962362736 0loss/SRqqt4/detection_loss‚1.514065068276202 /loss/FBt2/detection_loss‚3.1674254462122917 2loss/SRqqt1/assignment_loss‚1.4646316517499232 1loss/FBt3/assignment_loss‚0.24031831696628386 -loss/FRt2/detection_loss‚3.18828180556496 _step‚756 2loss/SRqqt2/assignment_loss‚2.1105644007523856 2loss/SRqqt4/assignment_loss‚1.0042463907910908 /loss/FRt4/detection_loss‚2.0952839748087255 /loss/FBt3/detection_loss‚2.9052107453346254 /loss/FRt3/assignment_loss‚2.433178201664326 0loss/FRt4/assignment_loss‚1.8306168968027288 0loss/SRqqt1/detection_loss‚2.192033895424434 /loss/FBt1/detection_loss‚4.1714844306310015 &loss/total_loss‚47.463361128093496 0loss/FBt1/assignment_loss‚0.7739133449892203 1loss/FBt4/assignment_loss‚0.14819937090837723 /loss/FBt4/detection_loss‚0.6178721718538505 epoch‚7 -loss/FRt3/detection_loss‚2.44280853313069 % _timestamp‚1.7884026103256147e+09 2loss/SRqqt3/assignment_loss‚1.3725103992605354 1loss/FBt2/assignment_loss‚0.22202992531741472 1loss/SRqqt3/detection_loss‚2.1873213216298963 trainer/global_step‚9199 0loss/FRt1/assignment_loss‚3.1543760310519824 /loss/FRt2/assignment_loss‚3.933777098854383 .loss/FRt1/detection_loss‚1.988779091835039 0loss/SRqqt2/detection_loss‚2.304432614283128ô:¥K‚˜#‚(:’ ¼·ãÔÞÇ”¯ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚147.487' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.497400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ú¨÷x™#‚(:í ¼·ãÔ•Ï° disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0* proc.memory.percent‚16.388988494873047 cpu‚25.35509069790435 disk.sda3.out‚ 8011.44921875 network.sent‚ 2.269066e+07* disk./.usagePercent‚15.670809220154355 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.dm-0.in‚ 399.828125& disk.nvme0n1.out‚35181.02197265625$ memory_percent‚96.00772857666016$ disk./.usageGB‚233.5063819885254 disk.dm-1.in‚ 0.06640625 disk.rbd0.out‚0 disk.sda1.in‚0 disk.rbd1.out‚ 18.0703125! proc.memory.rssMB‚ 2685.171875 disk.drbd2080.out‚0# disk.dm-0.out‚35181.02197265625( disk.nvme0c0n1.out‚35181.02197265625% proc.memory.availableMB‚ 654.09375 proc.cpu.threads‚17 network.recv‚ 2.559175e+06 disk.sda.out‚ 8011.44921875 disk.drbd2265.in‚0 disk.sda3.in‚ 0.06640625 disk.nvme0c0n1.in‚ 399.828125 disk.nvme0n1.in‚ 399.828125 disk.drbd1716.in‚0 disk.dm-1.out‚ 8011.44921875 disk.sda1.out‚0 disk.sda.in‚ 0.06640625 disk.sda2.in‚0 disk.rbd0.in‚08–I¼š#‚(:’ Ë·ãÔ®Ñì¯ gpu.0.gpu‚42.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚131.312' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.262400000000003 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö¦EÄt›#‚(:é Ë·ãÔò™¦±* disk./.usagePercent‚15.670813316274904 disk.drbd1716.in‚0 disk.dm-1.out‚ 8012.046875 disk.drbd2080.out‚0 disk.rbd0.in‚0* proc.memory.percent‚16.388988494873047 disk.sda1.out‚0( disk.nvme0c0n1.out‚35181.41552734375 disk.drbd2265.out‚0 disk.sda3.in‚ 0.06640625 disk.sda2.out‚0 disk.sda2.in‚0 disk.dm-1.in‚ 0.06640625% proc.memory.availableMB‚ 653.84375! proc.memory.rssMB‚ 2685.171875 network.sent‚ 2.2802028e+07 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 18.1328125 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 399.828125 disk.sda.out‚ 8012.046875 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.dm-0.in‚ 399.828125# disk.dm-0.out‚35181.41552734375 disk.sda3.out‚ 8012.046875 cpu‚25.354491870504138 network.recv‚ 2.571735e+06% disk./.usageGB‚233.50644302368164# memory_percent‚96.0092544555664 disk.nvme0n1.in‚ 399.828125& disk.nvme0n1.out‚35181.41552734375 disk.sda.in‚ 0.06640625 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0þ~}V2œ#- _wandbruntime‚7270  _runtime‚7270ÂÅNxÈ# ' lr-AdamW/pg1‚3.4179643201342474e-05 ' lr-AdamW/pg2‚3.4179643201342474e-05 trainer/global_step‚9249 % _timestamp‚1.7884026453937316e+09 _runtime‚ 7270.03060851 _step‚757Ùìo Íž#Ç ' lr-AdamW/pg1‚3.4179643201342474e-05 ' lr-AdamW/pg2‚3.4179643201342474e-05 trainer/global_step‚9249 % _timestamp‚1.7884026453937316e+09 _runtime‚ 7270.03060851 _step‚757õŠ ä2Ÿ#- _wandbruntime‚7270  _runtime‚7270Edéó§ #æO YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚260573815946240 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 123508417 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚260884029911058 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52396998 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚260573833949184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚247878898738176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186918334 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚260573833949184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚273616363719168 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 186938372 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚273616550722048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13042547772928 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13268220988928 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13004966051478 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52396463 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚289369353575936 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚276101132587008 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 % _timestamp‚1.7884026453944645e+09 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 308766881 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 123508954 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚289369588100608 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _runtime‚ 7270.03147826 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13042716772864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 186938372 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121828509 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121828507 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13005131172882 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121828509 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 60DeviceStatsMonitor.on_0 ñåEtrain_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 trainer/global_step‚9249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚273616363719168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚260573815946240 _step‚758 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13042716772864 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚537 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚276101156527104 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚247878881698816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13268431573504 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚260883847750294 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚273616550722048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71111954 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 308746841 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121828507 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71111956 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13042547772928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚535 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 186918334 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13042716772864 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 186938372 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 121828509 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 121828507 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13005131172882 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 121828509 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 trainer/global_step‚9249 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚273616363719168 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚260573815946240 _step‚758 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13042716772864 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚537 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚276101156527104 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚247878881698816 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13268431573504 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚260883847750294 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚273616550722048 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71111954 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 308746841 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 121828507 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71111956 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13042547772928 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚535 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 186918334 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 123508417 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚260884029911058 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52396998 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚260573833949184 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚247878898738176 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 186918334 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚260573833949184 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚273616363719168 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 186938372 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚273616550722048 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992öΟ2¢#- _wandbruntime‚7270  _runtime‚7270zoiá÷&£#ñM LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7884026460705242e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _step‚759 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13043940911104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52401872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚9249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚260601085310976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚289399685885952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚518 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13044109924352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52402388 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚7270.707929659 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚260911180097174 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13006520337634 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚273645213238272 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚247904841920512 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13044109924352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 308799353 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 186958185 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123521203 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 308779287 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚276130069475328 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚260911362258146 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 186958185 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 186938121 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121841166 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71119331 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚260601103313920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 308779287 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚260601103313920 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123521721 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26GÍ~!¤#öM M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚247904841920512 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13044109924352 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 308799353 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 186958185 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 123521203 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 308779287 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚276130069475328 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚260911362258146 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 186958185 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 186938121 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 121841166 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71119331 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚260601103313920 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 308779287 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚260601103313920 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 123521721 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7884026460705242e+09 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _step‚759 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13043940911104 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52401872 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚9249 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚260601085310976 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚289399685885952 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚518 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13044109924352 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52402388 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚7270.707929659 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚260911180097174 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13006520337634 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚273645213238272 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944÷5^s 2¥#- _wandbruntime‚7270  _runtime‚7270UâŸ[¦#Õ /loss/FRt2/detection_loss‚3.2590011610434604 1loss/SRqqt2/detection_loss‚2.6542870799700418 &loss/total_loss‚44.078723112440365 trainer/global_step‚9249 2loss/SRqqt2/assignment_loss‚1.7089155558496714 1loss/FBt1/assignment_loss‚0.07277062050984569 0loss/FBt4/assignment_loss‚0.1455220247168653 .loss/FBt1/detection_loss‚4.151906085014343 1loss/SRqqt4/detection_loss‚0.9570886108436083 epoch‚7 0loss/FRt2/assignment_loss‚3.1027720134991865 1loss/FBt2/assignment_loss‚0.33190187796832593 -loss/FRt1/detection_loss‚2.07856856191412 % _timestamp‚1.7884026460722163e+09 2loss/SRqqt4/assignment_loss‚1.2071842948513989 .loss/FRt4/detection_loss‚2.054505703088484 1loss/SRqqt3/detection_loss‚1.7375156602787751 /loss/FBt4/detection_loss‚0.7074684380657144 0loss/FBt3/assignment_loss‚0.6167397951441151 /loss/FBt3/detection_loss‚1.9991333654948644 _runtime‚7270.708980041 0loss/FRt3/assignment_loss‚2.8034611122948783 1loss/SRqqt1/assignment_loss‚1.372828747641216 1loss/SRqqt3/assignment_loss‚1.647236627423101 1loss/SRqqt1/detection_loss‚1.8461042132149352 _step‚760 0loss/FRt1/assignment_loss‚3.5073088851380856 0loss/FRt4/assignment_loss‚1.4790924865948527 /loss/FRt3/detection_loss‚2.4703418846641267 .loss/FBt2/detection_loss‚2.167068307216351ØKû‹`§#Ú 2loss/SRqqt4/assignment_loss‚1.2071842948513989 .loss/FRt4/detection_loss‚2.054505703088484 1loss/SRqqt3/detection_loss‚1.7375156602787751 /loss/FBt4/detection_loss‚0.7074684380657144 0loss/FBt3/assignment_loss‚0.6167397951441151 /loss/FBt3/detection_loss‚1.9991333654948644 _runtime‚7270.708980041 0loss/FRt3/assignment_loss‚2.8034611122948783 1loss/SRqqt1/assignment_loss‚1.372828747641216 1loss/SRqqt3/assignment_loss‚1.647236627423101 1loss/SRqqt1/detection_loss‚1.8461042132149352 _step‚760 0loss/FRt1/assignment_loss‚3.5073088851380856 0loss/FRt4/assignment_loss‚1.4790924865948527 /loss/FRt3/detection_loss‚2.4703418846641267 .loss/FBt2/detection_loss‚2.167068307216351 /loss/FRt2/detection_loss‚3.2590011610434604 1loss/SRqqt2/detection_loss‚2.6542870799700418 &loss/total_loss‚44.078723112440365 trainer/global_step‚9249 2loss/SRqqt2/assignment_loss‚1.7089155558496714 1loss/FBt1/assignment_loss‚0.07277062050984569 0loss/FBt4/assignment_loss‚0.1455220247168653 .loss/FBt1/detection_loss‚4.151906085014343 1loss/SRqqt4/detection_loss‚0.9570886108436083 epoch‚7 0loss/FRt2/assignment_loss‚3.1027720134991865 1loss/FBt2/assignment_loss‚0.33190187796832593 -loss/FRt1/detection_loss‚2.07856856191412 % _timestamp‚1.7884026460722163e+09øÏ_œ¨#‚(:‘ Ú·ãԭφ® gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚128.89' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.777999999999995 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0³{}©#‚(:ò Ú·ãÔ«ªù°$ disk./.usageGB‚233.5055274963379 disk.sda1.out‚0 disk.sda3.in‚ 0.06640625 disk.rbd1.in‚0 network.recv‚ 2.575285e+06 disk.sda1.in‚0$ memory_percent‚96.00803852081299 network.sent‚ 2.2807428e+07* disk./.usagePercent‚15.670751874466685 disk.sda2.out‚0 disk.drbd1716.out‚0( disk.nvme0c0n1.out‚35181.66943359375! proc.memory.rssMB‚ 2685.171875 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 399.828125 disk.sda3.out‚ 8012.52734375 disk.rbd1.out‚ 18.1328125( proc.memory.availableMB‚ 654.04296875* proc.memory.percent‚16.388988494873047 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.drbd1716.in‚0 disk.drbd2266.in‚0# disk.dm-0.out‚35181.66943359375 disk.nvme0n1.in‚ 399.828125 disk.sda.in‚ 0.06640625 disk.sda.out‚ 8012.52734375 cpu‚25.353834628594342 disk.dm-1.in‚ 0.06640625 disk.dm-1.out‚ 8012.52734375 disk.rbd0.in‚0& disk.nvme0n1.out‚35181.66943359375 disk.drbd2266.out‚0 disk.nvme0c0n1.in‚ 399.828125 proc.cpu.threads‚17 disk.drbd2265.in‚0œkçK’ª#‚(:‡ é·ãÔÿ€—® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚141.577' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.3154 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0“è `}«#‚(:ò é·ãÔ¹´±! proc.memory.rssMB‚ 2685.171875 proc.cpu.threads‚17 disk.rbd0.out‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2080.out‚0' proc.memory.availableMB‚ 654.2734375 cpu‚25.353242316988425 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚35182.23291015625 disk.nvme0n1.in‚ 399.828125 disk.sda.in‚ 0.06640625 disk.sda2.in‚0 disk.sda3.out‚ 8013.2890625 disk.rbd1.out‚ 18.2587890625& disk.nvme0n1.out‚35182.23291015625 disk.sda1.out‚0 disk.sda3.in‚ 0.06640625 disk.dm-1.out‚ 8013.2890625 network.sent‚ 2.2916438e+07 disk.sda.out‚ 8013.2890625 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0$ memory_percent‚96.00663185119629* disk./.usagePercent‚15.670750594429014 disk.drbd1716.in‚0 disk.dm-0.in‚ 399.828125# disk.dm-0.out‚35182.23291015625 disk.nvme0c0n1.in‚ 399.828125* proc.memory.percent‚16.388988494873047 disk.drbd1716.out‚0 network.recv‚ 2.582376e+06% disk./.usageGB‚233.50550842285156 disk.sda2.out‚0 disk.sda1.in‚0 disk.drbd2080.in‚0ܘc;’¬#‚(:‡ ø·ãԸ䴮 gpu.0.gpu‚32.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚134.263' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.8526 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0It½[€­#‚(:õ ø·ãÔË»í° disk.drbd1716.out‚0 disk.dm-1.out‚ 8013.73046875 disk.drbd2080.in‚0 disk.rbd0.out‚0 network.recv‚ 2.586362e+06( disk.nvme0c0n1.out‚35182.77392578125 disk.rbd1.out‚ 18.2587890625( proc.memory.availableMB‚ 654.02734375* proc.memory.percent‚16.388988494873047 cpu‚25.35259164674436 disk.nvme0c0n1.in‚ 399.828125 disk.sda.out‚ 8013.73046875 disk.drbd2080.out‚0 disk.sda2.out‚0 disk.rbd0.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17% disk./.usageGB‚233.50531005859375 disk.dm-1.in‚ 0.06640625 disk.drbd2266.in‚0 disk.rbd1.in‚0$ memory_percent‚96.00813388824463 disk.nvme0n1.in‚ 399.828125 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.dm-0.in‚ 399.828125 network.sent‚ 2.2922347e+07* disk./.usagePercent‚15.670737282037233 disk.sda.in‚ 0.06640625 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.06640625 disk.sda3.out‚ 8013.73046875& disk.nvme0n1.out‚35182.77392578125 disk.drbd2266.out‚0 disk.drbd2265.out‚0# disk.dm-0.out‚35182.77392578125! proc.memory.rssMB‚ 2685.171875§~¡Š2®#- _wandbruntime‚7305  _runtime‚7305n{ÕƯ#À & lr-AdamW/pg1‚3.305246152113162e-05 & lr-AdamW/pg2‚3.305246152113162e-05 trainer/global_step‚9299 $ _timestamp‚1.788402681313663e+09 _runtime‚7305.950518842 _step‚761Ît}þË°#Å & lr-AdamW/pg2‚3.305246152113162e-05 trainer/global_step‚9299 $ _timestamp‚1.788402681313663e+09 _runtime‚7305.950518842 _step‚761 & lr-AdamW/pg1‚3.305246152113162e-05ù?“©2±#- _wandbruntime‚7305  _runtime‚73057 ôí'²#çO 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚7305.951335593 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187931074 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚261937302185984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 124149666 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 % _timestamp‚1.7884026813143494e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 187911036 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122461459 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚515 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚249176040815616 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13112388095488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚277546803938304 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122461457 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚275049690281472 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚261937284183040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52668862 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 187931074 trainer/global_step‚9299 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122461457 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71480806 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚277546779998208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚290885940590592 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚261937302185984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 310392533 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚275049690281472 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13074437869886 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚249176057854976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _step‚762 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚275049503278592 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13074602991290 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13112219095552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚290886168823808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚261937284183040 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13339364885504 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204293120 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71480804 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 124150183 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚517 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13112219095552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52669377 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 310372493 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 187911036 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275049503278592 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚262250660846266 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122461459 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006žá³Ñò'³#ìO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚275049690281472 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13074437869886 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚249176057854976 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _step‚762 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚275049503278592 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13074602991290 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13112219095552 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚290886168823808 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚261937284183040 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13339364885504 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204293120 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚71480804 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 124150183 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚517 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13112219095552 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚52669377 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 310372493 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 187911036 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚275049503278592 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚262250660846266 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 122461459 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _runtime‚7305.951335593 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 187931074 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚261937302185984 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 124149666 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 % _timestamp‚1.7884026813143494e+09 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 187911036 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 122461459 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚515 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚249176040815616 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13112388095488 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚277546803938304 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 122461457 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚275049690281472 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚261937284183040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52668862 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 187931074 trainer/global_step‚9299 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 122461457 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚71480806 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚277546779998208 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚290885940590592 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚261937302185984 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 310392533 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0úÓ‹˜@2´#- _wandbruntime‚7306  _runtime‚7306K›í'× µ#ñM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚277575692946432 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚262277993193354 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 187930823 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13075827034430 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 310425005 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 310404939 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚249201983997952 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚261964571550720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71488181 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚249202001037312 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124162800 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13075992156042 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 187950887 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 @:DeviceStatsM.py onitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 122474116 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚290916520109568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13340590554624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚262277811032382 % _timestamp‚1.7884026819933856e+09 _step‚763 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚275078352797696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚277575716886528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 310425005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122474118 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 187950887 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13340803223040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13113612233728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52674096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71488183 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚7306.630783015 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚261964553547776 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122474116 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124162277 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚290916520109568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13340590554624 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚262277811032382 % _timestamp‚1.7884026819933856e+09 _step‚763 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚275078352797696 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚277575716886528 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 310425005 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 122474118 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 187950887 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13340803223040 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13113612233728 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52674096 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71488183 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚7306.630783015 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚261964553547776 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 122474116 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124162277 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 310425005 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 310404939 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚249201983997952 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚261964571550720 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚71488181 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚249202001037312 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 124162800 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13075992156042 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 187950887 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0û§ü 2·#- _wandbruntime‚7306  _runtime‚73069¦Ë¶Z¸#Ô 0loss/FRt2/assignment_loss‚3.4093241844421778 2loss/SRqqt3/assignment_loss‚1.6087748794299033 /loss/SRqqt2/detection_loss‚2.38185508052508 /loss/FBt4/assignment_loss‚0.134509846568711 0loss/SRqqt4/detection_loss‚1.174243335540478 /loss/FBt1/detection_loss‚2.2968501448631287 /loss/FBt3/detection_loss‚2.5134910482626696 _step‚764 0loss/FRt1/assignment_loss‚3.4096329274930453 /loss/FRt3/assignment_loss‚2.885519967973232 .loss/FBt2/detection_loss‚2.912749260663986 _runtime‚7306.632164722 2loss/SRqqt2/assignment_loss‚2.0657045925656954 0loss/FBt2/assignment_loss‚0.5608067624265516 1loss/FBt3/assignment_loss‚0.45511740215665625 1loss/SRqqt1/detection_loss‚1.8929232709937625 0loss/FRt4/assignment_loss‚1.4760841378625833 /loss/FRt1/detection_loss‚2.7595920445103395 /loss/FRt3/detection_loss‚2.3472096119076014 1loss/SRqqt3/detection_loss‚1.8794553806936298 /loss/FBt4/detection_loss‚0.7090024086527336 %loss/total_loss‚44.15248877238534 trainer/global_step‚9299 /loss/FRt4/detection_loss‚1.5537277313898195 2loss/SRqqt1/assignment_loss‚1.5440917927875286 2loss/SRqqt4/assignment_loss‚0.8440107309904236 /loss/FRt2/detection_loss‚2.7779010320321107 % _timestamp‚1.7884026819953964e+09 0loss/FBt1/assignment_loss‚0.5599111976534914 epoch‚7[ ´º_¹#Ù 0loss/FRt1/assignment_loss‚3.4096329274930453 /loss/FRt3/assignment_loss‚2.885519967973232 .loss/FBt2/detection_loss‚2.912749260663986 _runtime‚7306.632164722 2loss/SRqqt2/assignment_loss‚2.0657045925656954 0loss/FBt2/assignment_loss‚0.5608067624265516 1loss/FBt3/assignment_loss‚0.45511740215665625 1loss/SRqqt1/detection_loss‚1.8929232709937625 0loss/FRt4/assignment_loss‚1.4760841378625833 /loss/FRt1/detection_loss‚2.7595920445103395 /loss/FRt3/detection_loss‚2.3472096119076014 1loss/SRqqt3/detection_loss‚1.8794553806936298 /loss/FBt4/detection_loss‚0.7090024086527336 %loss/total_loss‚44.15248877238534 trainer/global_step‚9299 /loss/FRt4/detection_loss‚1.5537277313898195 2loss/SRqqt1/assignment_loss‚1.5440917927875286 2loss/SRqqt4/assignment_loss‚0.8440107309904236 /loss/FRt2/detection_loss‚2.7779010320321107 % _timestamp‚1.7884026819953964e+09 0loss/FBt1/assignment_loss‚0.5599111976534914 epoch‚7 0loss/FRt2/assignment_loss‚3.4093241844421778 2loss/SRqqt3/assignment_loss‚1.6087748794299033 /loss/SRqqt2/detection_loss‚2.38185508052508 /loss/FBt4/assignment_loss‚0.134509846568711 0loss/SRqqt4/detection_loss‚1.174243335540478 /loss/FBt1/detection_loss‚2.2968501448631287 /loss/FBt3/detection_loss‚2.5134910482626696 _step‚764ü³±¿n’º#‚(:‡ ‡¸ãÔ뇅® gpu.0.gpu‚33.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚132.667' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.5334 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÄÎÄÌr»#‚(:ç ‡¸ãÔâÒñ° network.sent‚ 2.2929075e+07 disk.dm-1.in‚ 0.06640625 disk.drbd2266.out‚0 disk.rbd1.out‚ 18.3212890625( proc.memory.availableMB‚ 654.03515625 cpu‚25.352007485788047* disk./.usagePercent‚15.670947464222845 disk.dm-1.out‚ 8014.26171875 disk.dm-0.out‚ 35187.015625 disk.sda1.in‚0# disk.nvme0c0n1.out‚ 35187.015625! disk.nvme0n1.out‚ 35187.015625* proc.memory.percent‚16.388988494873047 disk.drbd2266.in‚0 disk.rbd0.in‚0 network.recv‚ 2.591581e+06% disk./.usageGB‚233.50844192504883 disk.sda.out‚ 8014.26171875 disk.drbd2265.in‚0 disk.dm-0.in‚ 399.828125 disk.sda1.out‚0! proc.memory.rssMB‚ 2685.171875 disk.sda.in‚ 0.06640625 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.sda3.out‚ 8014.26171875 disk.nvme0c0n1.in‚ 399.828125 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.sda3.in‚ 0.06640625 disk.rbd1.in‚0$ memory_percent‚96.00808620452881 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.nvme0n1.in‚ 399.828125 proc.cpu.threads‚17 disk.drbd2080.in‚0~k’g’¼#‚(:‡ –¸ãÔÍ…ž® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚149.668' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.9336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œJ•Æa½#‚(:Ö –¸ãԢධ disk.rbd1.in‚0 disk.sda.out‚8014.75 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.dm-0.out‚ 35187.2578125& proc.memory.availableMB‚ 653.890625 disk.dm-0.in‚ 399.828125 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 399.828125 disk.dm-1.out‚8014.75 disk.rbd0.in‚0* proc.memory.percent‚16.388988494873047 cpu‚25.351395299410264* disk./.usagePercent‚15.670944904147502 disk.drbd1716.out‚0! proc.memory.rssMB‚ 2685.171875 disk.sda3.out‚8014.75 disk.nvme0n1.in‚ 399.828125$ memory_percent‚96.00896835327148 network.sent‚ 2.3036977e+07% disk./.usageGB‚233.50840377807617 disk.rbd1.out‚ 18.3212890625" disk.nvme0n1.out‚ 35187.2578125 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.sda.in‚ 0.06640625 disk.drbd2080.in‚0 disk.drbd2080.out‚0 network.recv‚ 2.596903e+06$ disk.nvme0c0n1.out‚ 35187.2578125 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.06640625%L -2¾#- _wandbruntime‚7341  _runtime‚7341 g/#É¿#à _step‚765 ' lr-AdamW/pg1‚3.1940793201506514e-05 ' lr-AdamW/pg2‚3.1940793201506514e-05 trainer/global_step‚9349 % _timestamp‚1.7884027168251998e+09 _runtime‚7341.462052684JDÔnÎÀ#È _runtime‚7341.462052684 _step‚765 ' lr-AdamW/pg1‚3.1940793201506514e-05 ' lr-AdamW/pg2‚3.1940793201506514e-05 trainer/global_step‚9349 % _timestamp‚1.7884027168251998e+09ý1ù[ë2Á#- _wandbruntime‚7341  _runtime‚7341Ü. ¥#Â#æO JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 188903738 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52938296 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13410085593088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚292402513002496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚263300770422784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚276482642838016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚292402747527168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚276482642838016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚263300752419840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13410296177664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚263300770422784 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 188903738 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚250473199932416 MGDeviceStatsMonitor.onçGµ°G_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188923776 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 188903738 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚52938296 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13410085593088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚292402513002496 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚263300770422784 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚276482642838016 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚292402747527168 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚276482642838016 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚263300752419840 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13410296177664 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚263300770422784 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 188903738 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚250473199932416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 188923776 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52944277 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚279021364297728 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124800770 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚263644441967590 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123107068 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 188943589 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71857033 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 312030591 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123107066 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚ 7342.1332764 _step‚767 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188923525 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13183283556352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚263328021784576 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚540 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 312030591 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13411502206464 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚263328039787520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13183452569600 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13411712777728 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188923525 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13145298852838 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚276511492357120 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 312050657 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13183452569600 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312050657 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52943739 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13183283556352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13145463974450 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚263644624128562 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚292432842564096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 % _timestamp‚1.7884027174959025e+09[?lxú&Æ#ôM E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚263328039787520 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13183452569600 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13411712777728 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 188923525 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13145298852838 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚276511492357120 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 312050657 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13183452569600 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 312050657 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚52943739 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13183283556352 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13145463974450 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚263644624128562 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚292432842564096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 % _timestamp‚1.7884027174959025e+09 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚52944277 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚279021364297728 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 124800770 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚263644441967590 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123107068 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 188943589 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚71857033 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 312030591 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123107066 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 _runtime‚ 7342.1332764 _step‚767 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 188923525 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13183283556352 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚263328021784576 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚540 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 312030591 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13411502206464ÿ© ÉH2Ç#- _wandbruntime‚7342  _runtime‚7342aÂDÈ#Ö 2loss/SRqqt2/assignment_loss‚2.0558343297905393 0loss/SRqqt1/detection_loss‚1.934424322775819 /loss/FBt1/detection_loss‚2.5518720924854277 0loss/FRt1/assignment_loss‚3.2732013007427785 2loss/SRqqt1/assignment_loss‚1.2438141302290289 1loss/SRqqt3/assignment_loss‚1.792846781667322 1loss/FBt3/assignment_loss‚0.12289658512178324 /loss/FRt3/detection_loss‚2.4707224590656085 .loss/FRt4/detection_loss‚1.749548010701357 1loss/SRqqt4/detection_loss‚1.1824622182757594 % _timestamp‚1.7884027174975185e+09 _step‚768 1loss/FBt1/assignment_loss‚0.32564872628690866 1loss/SRqqt2/detection_loss‚2.2424292696846857 trainer/global_step‚9349 1loss/FBt4/assignment_loss‚0.12238788858497666 /loss/FBt2/detection_loss‚2.9454866184128656 /loss/FBt4/detection_loss‚0.7490523344526688 epoch‚7 .loss/FRt1/detection_loss‚2.244511697520601 /loss/FRt2/detection_loss‚3.3071954172590505 .loss/FBt3/detection_loss‚2.907438202337785 &loss/total_loss‚44.121405749476246 /loss/FRt2/assignment_loss‚3.431398972221043 /loss/FRt4/assignment_loss‚1.350126331628755 1loss/FBt2/assignment_loss‚0.14584483021725544 1loss/SRqqt3/detection_loss‚1.7328527085483074 0loss/FRt3/assignment_loss‚3.2799947399359484 2loss/SRqqt4/assignment_loss‚0.9594157815299695 ¨„˜b_runtime‚ 7342.13429969úS0•aÉ#Û &loss/total_loss‚44.121405749476246 /loss/FRt2/assignment_loss‚3.431398972221043 /loss/FRt4/assignment_loss‚1.350126331628755 1loss/FBt2/assignment_loss‚0.14584483021725544 1loss/SRqqt3/detection_loss‚1.7328527085483074 0loss/FRt3/assignment_loss‚3.2799947399359484 2loss/SRqqt4/assignment_loss‚0.9594157815299695 _runtime‚ 7342.13429969 2loss/SRqqt2/assignment_loss‚2.0558343297905393 0loss/SRqqt1/detection_loss‚1.934424322775819 /loss/FBt1/detection_loss‚2.5518720924854277 0loss/FRt1/assignment_loss‚3.2732013007427785 2loss/SRqqt1/assignment_loss‚1.2438141302290289 1loss/SRqqt3/assignment_loss‚1.792846781667322 1loss/FBt3/assignment_loss‚0.12289658512178324 /loss/FRt3/detection_loss‚2.4707224590656085 .loss/FRt4/detection_loss‚1.749548010701357 1loss/SRqqt4/detection_loss‚1.1824622182757594 % _timestamp‚1.7884027174975185e+09 _step‚768 1loss/FBt1/assignment_loss‚0.32564872628690866 1loss/SRqqt2/detection_loss‚2.2424292696846857 trainer/global_step‚9349 1loss/FBt4/assignment_loss‚0.12238788858497666 /loss/FBt2/detection_loss‚2.9454866184128656 /loss/FBt4/detection_loss‚0.7490523344526688 epoch‚7 .loss/FRt1/detection_loss‚2.244511697520601 /loss/FRt2/detection_loss‚3.3071954172590505 .loss/FBt3/detection_loss‚2.907438202337785€—uºÊ#‚(:’ ¥¸ãÔ¹¼® gpu.0.gpu‚44.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚133.733' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.746599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ'å–Ë#‚(:ô ¥¸ãÔ‹¬‘² disk.sda3.in‚ 0.06640625 disk.sda3.out‚ 8015.51171875 disk.nvme0c0n1.in‚ 399.828125& disk.nvme0n1.out‚35188.07958984375$ memory_percent‚96.01681232452393 proc.cpu.threads‚17 disk.sda1.out‚0* proc.memory.percent‚16.388988494873047% disk./.usageGB‚233.50938034057617 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 18.447265625 disk.drbd1716.in‚0 disk.dm-1.out‚ 8015.51171875( proc.memory.availableMB‚ 652.60546875 network.sent‚ 2.3046448e+07 disk.drbd2265.out‚0# disk.dm-0.out‚35188.07958984375 cpu‚25.350736186727396 network.recv‚ 2.607816e+06 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0 disk.sda1.in‚0 disk.sda.in‚ 0.06640625 disk.sda2.in‚0( disk.nvme0c0n1.out‚35188.07958984375 disk.sda.out‚ 8015.51171875 disk.sda2.out‚0! proc.memory.rssMB‚ 2685.171875) disk./.usagePercent‚15.67101044207627 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 399.828125 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.06640625 disk.rbd0.in‚0 disk.dm-0.in‚ 399.828125—¬ÑpÌ#‚(:’ ´¸ãÔŠž® gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚35.0 gpu.0.powerWatts‚156.155' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚31.230999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0-°,|Í#‚(:ñ ´¸ãÔøžª± disk.rbd1.out‚ 18.533203125 network.recv‚ 2.614469e+06 disk.drbd2080.in‚0# disk.dm-0.out‚35189.03076171875 disk.nvme0n1.in‚ 399.828125% proc.memory.availableMB‚ 654.15625! proc.memory.rssMB‚ 2685.171875 proc.cpu.threads‚17 network.sent‚ 2.3155282e+07 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚ 0.06640625 disk.sda.out‚ 8016.16015625 disk.nvme0c0n1.in‚ 399.828125 disk.drbd2266.in‚0 disk.sda3.out‚ 8016.16015625* proc.memory.percent‚16.388988494873047* disk./.usagePercent‚15.671009162038596% disk./.usageGB‚233.50936126708984 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0& disk.nvme0n1.out‚35189.03076171875 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.06640625 disk.rbd0.in‚0( disk.nvme0c0n1.out‚35189.03076171875 disk.dm-1.out‚ 8016.16015625 disk.sda1.in‚0 disk.sda.in‚ 0.06640625# memory_percent‚96.0073471069336 cpu‚25.350063906855123 disk.sda2.out‚0 disk.rbd0.out‚0 disk.dm-0.in‚ 399.828125 disk.rbd1.in‚0Ä·Å2Î#- _wandbruntime‚7376  _runtime‚7376zŒqÉÏ#à ' lr-AdamW/pg1‚3.0844869912478665e-05 ' lr-AdamW/pg2‚3.0844869912478665e-05 trainer/global_step‚9399 % _timestamp‚1.7884027522480462e+09 _runtime‚7376.884922441 _step‚769 i;.ÎÐ#È trainer/global_step‚9399 % _timestamp‚1.7884027522480462e+09 _runtime‚7376.884922441 _step‚769 ' lr-AdamW/pg1‚3.0844869912478665e-05 ' lr-AdamW/pg2‚3.0844869912478665e-05\Ç÷\2Ñ#- _wandbruntime‚7376  _runtime‚7376Ad]í'Ò#çO SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13213381506702 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13251561740800 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 313623797 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚277915782397440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 189896440 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125427688 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚251770359049216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13251561740800 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 189916478 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚264664220656640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚277915969400320 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72218504 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 125428209 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13251730740736 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚251770376088576 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚264983740555918 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚277915782397440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚521 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53209703 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚9399 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123727357 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 % _timestamp‚1.7884027522487562e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚280438074820608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚264983922716682 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚264664220656640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 313643837 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53209184 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 189896440 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13251730740736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123727357 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13480947949568 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚264664238659584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚519 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _step‚770 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0g éò'Ó#ìO YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 189916478 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚264664220656640 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚277915969400320 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72218504 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 125428209 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13251730740736 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚251770376088576 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚264983740555918 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚277915782397440 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚521 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53209703 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚9399 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 123727357 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 % _timestamp‚1.7884027522487562e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚280438074820608 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚264983922716682 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚264664220656640 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 313643837 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53209184 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 189896440 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13251730740736 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 123727357 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13480947949568 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚264664238659584 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚519 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _step‚770 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13213381506702 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13251561740800 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 313623797 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚277915782397440 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 189896440 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 125427688 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚251770359049216 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13251561740800‚U<‘2Ô#- _wandbruntime‚7377  _runtime‚7377åù³äÕ#ñM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚293949587990016 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13482365709824 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123740018 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 313656243 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 189916227 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123740016 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚277944631916544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚293949353478656 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚7377.547245435 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 313656243 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚280466987768832 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚280467011708928 % _timestamp‚1.7884027529097419e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚264691508024320 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13214935792858 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13252954878976 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 125441372 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53214989 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚277944444900352 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚264691490021376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 189936291 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚500 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚277944631916544 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123740018 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚265011255063770 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚251796319270912 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313676309 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125440870 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9399 _step‚771 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚502 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.púQî eak‚ 104436224 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13214770671246 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 189936291 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123740016 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13252954878976 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72225881 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251796302231552 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13253123892224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53215489ü&Ö#öM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚277944631916544 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚293949353478656 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 _runtime‚7377.547245435 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 313656243 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚280466987768832 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚280467011708928 % _timestamp‚1.7884027529097419e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚264691508024320 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13214935792858 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13252954878976 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 125441372 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53214989 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚277944444900352 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚264691490021376 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 189936291 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚500 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚277944631916544 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 123740018 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚265011255063770 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚251796319270912 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 313676309 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 125440870 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚9399 _step‚771 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚502 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13214770671246 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 189936291 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 123740016 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13252954878976 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72225881 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚251796302231552 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13253123892224 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53215489 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚293949587990016 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13482365709824 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 123740018 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 313656243 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 189916227 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 123740016 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217ƒd"¥Û2×#- _wandbruntime‚7377  _runtime‚7377«©ŽbYØ#Ó /loss/FRt3/assignment_loss‚2.690639084293729 1loss/SRqqt1/assignment_loss‚1.260794636849314 .loss/FRt1/detection_loss‚2.081843677976075 0loss/SRqqt2/detection_loss‚2.740489140704826 1loss/SRqqt3/detection_loss‚1.7908873102881693 trainer/global_step‚9399 2loss/SRqqt4/assignment_loss‚1.2847271667118987 1loss/FBt2/assignment_loss‚0.33951436195981544 /loss/FRt2/detection_loss‚2.9430310581669663 _runtime‚7377.548238618 _step‚772 0loss/FBt3/assignment_loss‚0.8959421037825775 /loss/FBt3/detection_loss‚3.0001654284340993 0loss/SRqqt1/detection_loss‚1.731774240732193 1loss/SRqqt4/detection_loss‚1.2960100527852774 .loss/FBt2/detection_loss‚3.209010111434119 %loss/total_loss‚44.45229259223143 epoch‚7 /loss/FRt2/assignment_loss‚4.173641060337876 /loss/FRt4/assignment_loss‚1.379304790496826 1loss/FBt1/assignment_loss‚0.17161604995591612 /loss/FBt1/detection_loss‚2.2427550077438356 0loss/FRt1/assignment_loss‚2.6419549280946906 2loss/SRqqt2/assignment_loss‚2.0460988003070706 1loss/SRqqt3/assignment_loss‚1.349811583689668 1loss/FBt4/assignment_loss‚0.21502067752426118 /loss/FBt4/detection_loss‚0.6212801083803494 % _timestamp‚1.7884027529114695e+09 /loss/FRt3/detection_loss‚2.5901130877789997 /loss/FRt4/detection_loss‚1.7558681238028737§šØ.^Ù#Ø 0loss/FRt1/assignment_loss‚2.6419549280946906 2loss/SRqqt2/assignment_loss‚2.0460988003070706 1loss/SRqqt3/assignment_loss‚1.349811583689668 1loss/FBt4/assignment_loss‚0.21502067752426118 /loss/FBt4/detection_loss‚0.6212801083803494 % _timestamp‚1.7884027529114695e+09 /loss/FRt3/detection_loss‚2.5901130877789997 /loss/FRt4/detection_loss‚1.7558681238028737 /loss/FRt3/assignment_loss‚2.690639084293729 1loss/SRqqt1/assignment_loss‚1.260794636849314 .loss/FRt1/detection_loss‚2.081843677976075 0loss/SRqqt2/detection_loss‚2.740489140704826 1loss/SRqqt3/detection_loss‚1.7908873102881693 trainer/global_step‚9399 2loss/SRqqt4/assignment_loss‚1.2847271667118987 1loss/FBt2/assignment_loss‚0.33951436195981544 /loss/FRt2/detection_loss‚2.9430310581669663 _runtime‚7377.548238618 _step‚772 0loss/FBt3/assignment_loss‚0.8959421037825775 /loss/FBt3/detection_loss‚3.0001654284340993 0loss/SRqqt1/detection_loss‚1.731774240732193 1loss/SRqqt4/detection_loss‚1.2960100527852774 .loss/FBt2/detection_loss‚3.209010111434119 %loss/total_loss‚44.45229259223143 epoch‚7 /loss/FRt2/assignment_loss‚4.173641060337876 /loss/FRt4/assignment_loss‚1.379304790496826 1loss/FBt1/assignment_loss‚0.17161604995591612 /loss/FBt1/detection_loss‚2.2427550077438356„†ÜÞ’Ú#‚(:‡ øãÔ¤—·® gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚137.511' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.5022 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œÉ¸ïxÛ#‚(:í øãÔìø­² disk.sda.in‚ 0.06640625 disk.drbd2080.in‚0 disk.drbd2080.out‚0$ memory_percent‚96.01552486419678( proc.memory.availableMB‚ 652.81640625 proc.cpu.threads‚17$ disk./.usageGB‚233.5094985961914 disk.nvme0c0n1.in‚ 399.828125 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.sda3.out‚ 8016.80078125! proc.memory.rssMB‚ 2685.171875 disk.rbd1.in‚0 disk.rbd0.out‚0 disk.sda1.in‚0 network.sent‚ 2.3168582e+07 disk.rbd1.out‚ 18.595703125$ disk.nvme0n1.out‚35190.470703125 disk.sda.out‚ 8016.80078125 disk.sda2.out‚0 cpu‚25.349448688260328 disk.drbd2265.out‚0! disk.dm-0.out‚35190.470703125 network.recv‚ 2.63254e+06 disk.nvme0n1.in‚ 399.828125 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.06640625 disk.sda1.out‚0 disk.sda3.in‚ 0.06640625* disk./.usagePercent‚15.671018378309828& disk.nvme0c0n1.out‚35190.470703125 disk.dm-1.out‚ 8016.80078125 disk.drbd2266.out‚0 disk.rbd0.in‚0* proc.memory.percent‚16.388988494873047 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.dm-0.in‚ 399.828125H½Ä Ü#‚(:’ Ò¸ãÔé ® gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚145.065' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.012999999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ªYŠ yÝ#‚(:î Ò¸ãÔ·ž±± disk.sda.out‚ 8017.43359375 disk.sda2.in‚0* disk./.usagePercent‚15.671014538196815 disk.drbd2265.out‚0 disk.sda1.out‚0$ disk.nvme0n1.out‚35190.869140625 disk.drbd2080.in‚0 disk.rbd0.in‚0! disk.dm-0.out‚35190.869140625 disk.sda1.in‚0 disk.rbd1.in‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 653.9140625 disk.dm-1.out‚ 8017.43359375 disk.drbd2265.in‚0 disk.rbd1.out‚ 18.595703125! proc.memory.rssMB‚ 2685.171875% disk./.usageGB‚233.50944137573242 disk.dm-1.in‚ 0.06640625 disk.drbd2266.in‚0 disk.rbd0.out‚0 disk.nvme0c0n1.in‚ 399.828125 network.sent‚ 2.3276667e+07 disk.drbd2080.out‚0 disk.sda3.in‚ 0.06640625 disk.sda.in‚ 0.06640625$ memory_percent‚96.00882530212402 cpu‚25.348816606114333 proc.cpu.threads‚17 disk.sda3.out‚ 8017.43359375 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.dm-0.in‚ 399.828125& disk.nvme0c0n1.out‚35190.869140625 disk.drbd1716.in‚0* proc.memory.percent‚16.388988494873047 network.recv‚ 2.637918e+06 disk.nvme0n1.in‚ 399.828125€èì^Þ#‚(:’ á¸ãÔþÚ¥® gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚129.949' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.989800000000002 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0±SRyß#‚(:î á¸ãÔãúñ° network.sent‚ 2.3285862e+07 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.06640625 disk.dm-1.out‚ 8018.078125 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd0.in‚0( disk.nvme0c0n1.out‚35191.70654296875 disk.nvme0n1.in‚ 399.828125 disk.sda.out‚ 8018.078125 cpu‚25.34818355385451 disk.sda.in‚ 0.06640625 disk.drbd1716.in‚0 disk.drbd2080.out‚0$ memory_percent‚96.01328372955322! proc.memory.rssMB‚ 2685.171875 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.dm-0.in‚ 399.828125# disk.dm-0.out‚35191.70654296875 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.rbd1.in‚0 disk.sda1.out‚0* disk./.usagePercent‚15.670992777556405% disk./.usageGB‚233.50911712646484 disk.rbd1.out‚ 18.7216796875 disk.nvme0c0n1.in‚ 399.828125 disk.sda2.out‚0 network.recv‚ 2.64902e+06 disk.sda3.in‚ 0.06640625 disk.sda3.out‚ 8018.078125& disk.nvme0n1.out‚35191.70654296875 disk.rbd0.out‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 653.18359375* proc.memory.percent‚16.388988494873047L/o(2à#- _wandbruntime‚7412  _runtime‚7412 Ub Éá#à ' lr-AdamW/pg1‚2.9764920042818937e-05 ' lr-AdamW/pg2‚2.9764920042818937e-05 trainer/global_step‚9449 % _timestamp‚1.7884027873706377e+09 _runtime‚7412.007460516 _step‚773XPÎâ#È ' lr-AdamW/pg1‚2.9764920042818937e-05 ' lr-AdamW/pg2‚2.9764920042818937e-05 trainer/global_step‚9449 % _timestamp‚1.7884027873706377e+09 _runtime‚7412.007460516 _step‚773…8Xöb2ã#- _wandbruntime‚7412  _runtime‚7412¨+ÂJí'ä#çO BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 126067597 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53480791 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚266027706896384 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53480243 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 315269489 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚253067535205376 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124360309 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13321233063424 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 190889142 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 190909180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221070336 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13283018446514 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124360307 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚266027706896384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 315249449 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279348921956864 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚266027688893440 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚266350553651890 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72587356 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245010432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13282853325110 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72587354 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚281883722231808 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _step‚774I;7så#ìO @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 315249449 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚279348921956864 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚266027688893440 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚266350553651890 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚72587356 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245010432 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13282853325110 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72587354 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚281883722231808 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _step‚774 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 126067597 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53480791 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚266027706896384 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚53480243 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 315269489 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚253067535205376 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124360309 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13321233063424 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 190889142 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 190909180 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 221070336 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13283018446514 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124360307 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚266027706896384 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚548 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160†¤Á]½2æ#- _wandbruntime‚7412  _runtime‚7412v75÷&ç#ñM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚253093478387712 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚266054976261120 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚281912659120128 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13553255390208 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 124372966 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚295466122984448 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 126080917 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 190928993 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124372966 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315301961 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72594733 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53486184 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13284407611266 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124372968 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124372968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚279377584459776 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 315281895 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 _runtime‚7412.670659696 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 315301961 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 190928993 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 190908929 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 trainer/global_step‚9449 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72594731 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚266377885998978 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 % _timestamp‚1.7884027880333016e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 190908929 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚279377771475968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚266054958258176 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126080380 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13284242489654 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚279377771475968 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13322795214848 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚266054976261120 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚266054958258176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 124372966 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 315301961 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚72594733 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚53486184 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13284407611266 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 124372968 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 124372968 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚279377584459776 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 315281895 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 _runtime‚7412.670659696 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 315301961 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 190928993 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 190908929 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 trainer/global_step‚9449 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72594731 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚266377885998978 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 % _timestamp‚1.7884027880333016e+09 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 190908929 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚279377771475968 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚266054958258176 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126080380 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13284242489654 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚279377771475968 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13322795214848 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚266054976261120 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚266054958258176 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 EDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124993257 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _step‚778 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124993257 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚254364694322176 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13391073385984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72956204 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚7447.707039079 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚267391157130240 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 191901882 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 191881844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191881844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚267391175133184 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚280782061516288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚267717002426334 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚280782061516288 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚296952331784192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226136064 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126708935 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚280782248519168 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124993259 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53752729 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚267717184587098 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚280782248519168 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚283329369643008 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13352325143518 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 316875101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13390904386048 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 126708397 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0»2\Ëò'õ#ìO aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚283329369643008 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13352325143518 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 316875101 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13390904386048 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 126708397 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13391073385984 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 124993259 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚296952105648128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202195968 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 trainer/global_step‚9499 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13622938201088 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚267391157130240 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚536 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 124993257 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _step‚778 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 124993257 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚254364694322176 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13391073385984 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚72956204 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚7447.707039079 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚267391157130240 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 191901882 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 191881844 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 191881844 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚267391175133184 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚280782061516288 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚267717002426334 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚280782061516288 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚296952331784192 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226136064 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 126708935 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚280782248519168 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚538 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 124993259 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚53752729 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚267717184587098 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚280782248519168Š™'vb2ö#- _wandbruntime‚7448  _runtime‚7448oW>°&÷#ñM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 191921695 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚296982665524736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚283358306531328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚7448.381350814 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125005916 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 316907547 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚9499 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚280810911035392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 316927613 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚267744516934186 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚254390620465152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚280810911035392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚283358282591232 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚254390637504512 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125005918 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 316927613 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 % _timestamp‚1.7884028237440014e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚267418426494976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13624358993408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72963581 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚267744334773214 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 126721701 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 316907547 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125005916 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 191921695 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 _step‚779 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚267418444497920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13392466537472 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13392297524224 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 191901631 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53757595 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125005918 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚267418426494976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126721176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13353714308062 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13392466537472 HA ÐG@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525呈Eü&ø#öM KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 126721176 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13353714308062 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13392466537472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 191921695 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚296982665524736 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚283358306531328 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 _runtime‚7448.381350814 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125005916 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 316907547 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 trainer/global_step‚9499 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚280810911035392 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 316927613 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚267744516934186 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚254390620465152 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚280810911035392 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚283358282591232 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚254390637504512 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125005918 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 316927613 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 % _timestamp‚1.7884028237440014e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚267418426494976 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13624358993408 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚72963581 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚267744334773214 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 126721701 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 316907547 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125005916 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 191921695 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 _step‚779 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚267418444497920 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13392466537472 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13392297524224 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 191901631 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚53757595 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125005918 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚267418426494976 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160‹|‹ûÙ2ù#- _wandbruntime‚7448  _runtime‚7448“BšUú#Ï /loss/FRt2/assignment_loss‚3.843872527281443 2loss/SRqqt1/assignment_loss‚1.3368525692060882 0loss/FBt4/assignment_loss‚0.1667664185986979 /loss/SRqqt2/detection_loss‚2.62158342288888 .loss/FBt1/detection_loss‚3.540210497379303 epoch‚7 /loss/FRt4/assignment_loss‚2.550116004049778 /loss/FRt2/detection_loss‚3.1518661454319954 .loss/FRt3/detection_loss‚2.333518460046413 1loss/SRqqt4/detection_loss‚1.4169143066905878 _step‚780 2loss/SRqqt2/assignment_loss‚2.0741496112035667 1loss/FBt1/assignment_loss‚0.36579802455307797 /loss/FBt2/detection_loss‚2.3446507189008923 $loss/total_loss‚46.9345672167618 0loss/FRt3/assignment_loss‚3.3889643411303676 1loss/SRqqt3/assignment_loss‚1.288004920217726 1loss/SRqqt3/detection_loss‚1.5869898121390078 /loss/FBt4/detection_loss‚0.6426491974412489 _runtime‚7448.382361991 $ _timestamp‚1.788402823745617e+09 0loss/FBt3/assignment_loss‚0.1068612998349181 0loss/FBt2/assignment_loss‚0.2512006714109955 .loss/FRt4/detection_loss‚2.197399486415088 1loss/SRqqt1/detection_loss‚1.6598091899764305 .loss/FBt3/detection_loss‚3.036280487264906 trainer/global_step‚9499 0loss/FRt1/assignment_loss‚3.6562417113780974 2loss/SRqqt4/assignment_loss‚1.2319418611004949 /loss/FRt1/detection_loss‚2.1419255322217947X°/eZû#Ô 0loss/FBt4/assignment_loss‚0.1667664185986979 /loss/SRqqt2/detection_loss‚2.62158342288888 .loss/FBt1/detection_loss‚3.540210497379303 epoch‚7 /loss/FRt4/assignment_loss‚2.550116004049778 /loss/FRt2/detection_loss‚3.1518661454319954 .loss/FRt3/detection_loss‚2.333518460046413 1loss/SRqqt4/detection_loss‚1.4169143066905878 _step‚780 2loss/SRqqt2/assignment_loss‚2.0741496112035667 1loss/FBt1/assignment_loss‚0.36579802455307797 /loss/FBt2/detection_loss‚2.3446507189008923 $loss/total_loss‚46.9345672167618 0loss/FRt3/assignment_loss‚3.3889643411303676 1loss/SRqqt3/assignment_loss‚1.288004920217726 1loss/SRqqt3/detection_loss‚1.5869898121390078 /loss/FBt4/detection_loss‚0.6426491974412489 _runtime‚7448.382361991 $ _timestamp‚1.788402823745617e+09 0loss/FBt3/assignment_loss‚0.1068612998349181 0loss/FBt2/assignment_loss‚0.2512006714109955 .loss/FRt4/detection_loss‚2.197399486415088 1loss/SRqqt1/detection_loss‚1.6598091899764305 .loss/FBt3/detection_loss‚3.036280487264906 trainer/global_step‚9499 0loss/FRt1/assignment_loss‚3.6562417113780974 2loss/SRqqt4/assignment_loss‚1.2319418611004949 /loss/FRt1/detection_loss‚2.1419255322217947 /loss/FRt2/assignment_loss‚3.843872527281443 2loss/SRqqt1/assignment_loss‚1.3368525692060882Œ§0¢É’ü#‚(:‡ Ž¹ãÔ¹øû­ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚150.197' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚30.0394 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0˲b€ý#‚(:õ Ž¹ãÔ•ˆù°! proc.memory.rssMB‚ 2685.171875* proc.memory.percent‚16.388988494873047 disk.drbd1716.out‚0 disk.dm-1.out‚ 8019.80078125 network.sent‚ 2.340966e+07 disk.sda1.in‚0 disk.sda3.in‚ 0.06640625 disk.sda3.out‚ 8019.80078125 disk.nvme0n1.in‚ 399.828125 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚0( disk.nvme0c0n1.out‚35192.65576171875 disk.dm-1.in‚ 0.06640625 disk.drbd2266.in‚0# disk.dm-0.out‚35192.65576171875 disk.nvme0c0n1.in‚ 399.828125& disk.nvme0n1.out‚35192.65576171875$ memory_percent‚96.00903987884521( proc.memory.availableMB‚ 653.87890625 disk.sda.out‚ 8019.80078125 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.rbd0.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17% disk./.usageGB‚233.50906372070312 disk.sda.in‚ 0.06640625 disk.sda2.in‚0 disk.dm-0.in‚ 399.828125 disk.sda1.out‚0 cpu‚25.346355947973123 network.recv‚ 2.669963e+06* disk./.usagePercent‚15.670989193450927 disk.drbd2266.out‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.rbd1.out‚ 18.8701171875[’èœþ#‚(:‘ ¹ãÔ¨€ü­ gpu.0.gpu‚1.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚149.908' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.981599999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;Mä~ÿ#‚(:ó ¹ãÔâÜ¿± network.recv‚ 2.676465e+06 disk.sda3.out‚ 8020.30078125 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.dm-0.in‚ 399.828125! proc.memory.rssMB‚ 2685.171875 cpu‚25.345758911120257 network.sent‚ 2.3518605e+07 disk.rbd1.in‚0 disk.drbd2080.in‚0# disk.dm-0.out‚35193.13623046875 disk.sda.in‚ 0.06640625$ memory_percent‚96.01070880889893 disk.drbd2080.out‚0 disk.nvme0c0n1.in‚ 399.828125( disk.nvme0c0n1.out‚35193.13623046875 disk.dm-1.in‚ 0.06640625 proc.cpu.threads‚17 disk.sda3.in‚ 0.06640625 disk.sda.out‚ 8020.30078125 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 653.60546875$ disk./.usageGB‚233.5090103149414 disk.nvme0n1.in‚ 399.828125& disk.nvme0n1.out‚35193.13623046875* proc.memory.percent‚16.388988494873047 disk.rbd1.out‚ 18.99609375 disk.sda2.in‚0 disk.rbd0.in‚0 disk.rbd0.out‚0* disk./.usagePercent‚15.670985609345447 disk.drbd1716.out‚0 disk.dm-1.out‚ 8020.30078125 disk.drbd2266.in‚0 disk.drbd2266.out‚0¥V’2€$- _wandbruntime‚7483  _runtime‚7483ô·éÈ$ ' lr-AdamW/pg1‚2.7653837425601768e-05 ' lr-AdamW/pg2‚2.7653837425601768e-05 trainer/global_step‚9549 $ _timestamp‚1.788402858527932e+09 _runtime‚7483.164797893 _step‚781ƒ¤zŒÍ‚$Ç _step‚781 ' lr-AdamW/pg1‚2.7653837425601768e-05 ' lr-AdamW/pg2‚2.7653837425601768e-05 trainer/global_step‚9549 $ _timestamp‚1.788402858527932e+09 _runtime‚7483.164797893Ñ! N2ƒ$- _wandbruntime‚7483  _runtime‚7483lN+‚ì'„$æO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚298468652439552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚284775040994304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚284775017054208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 192894584 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127350128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13460744708608 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚282215201075712 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚282215388078592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125626207 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 192874546 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚282215388078592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73325054 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚282215201075712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚268754625367040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13693635385344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚268754643369984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54024530 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192894584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54025072 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚269083815522306 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚268754625367040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13421796961926 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚9549 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚268754643369984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125626209 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217J¾$b…$ëO YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚282215388078592 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 125626207 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 192874546 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚282215388078592 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73325054 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚282215201075712 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚268754625367040 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13693635385344 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚268754643369984 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54024530 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 192894584 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54025072 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚269083815522306 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚268754625367040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13421796961926 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚9549 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚268754643369984 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 125626209 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚298468652439552 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚284775040994304 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚284775017054208 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 192894584 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127350128 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13460744708608 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚282215201075712ŽM¸ ‘2†$- _wandbruntime‚7483  _runtime‚7483Q&‡å÷&‡$ñM >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 _step‚783 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318553265 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 318533199 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54030501 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13461968846848 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚284803953942528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚284803930002432 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 318553265 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 192914397 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚255687796621312 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚255687779581952 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13423186126470 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73332431 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54029964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125638866 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127362934 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13423351248082 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7483.837370586 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚268781912734720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125638868 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚269110965708422 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125638868 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127362395 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚268781894731776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13462137860096 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125638866 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13462137860096 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚282244050594816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13695045260800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚268781912734720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 192894333 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 % _timestamp‚1.7884028592001026e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73332433 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192914397 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚268781894731776 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13461968846848 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚298499218163200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚269111147869394 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612ŠDfÃü&ˆ$öM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚73332433 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 192914397 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚268781894731776 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13461968846848 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚298499218163200 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚269111147869394 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 _step‚783 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 318553265 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 318533199 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54030501 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13461968846848 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚284803953942528 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚284803930002432 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 318553265 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 192914397 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚255687796621312 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚255687779581952 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13423186126470 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73332431 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54029964 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 125638866 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 127362934 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13423351248082 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7483.837370586 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚268781912734720 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 125638868 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚269110965708422 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 125638868 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 127362395 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚268781894731776 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13462137860096 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 125638866 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13462137860096 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚282244050594816 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13695045260800 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚268781912734720 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 192894333 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 % _timestamp‚1.7884028592001026e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217¨-*2‰$- _wandbruntime‚7483  _runtime‚7483”Š‘îVŠ$Ð 1loss/SRqqt3/detection_loss‚1.7300443451512943 .loss/FBt2/detection_loss‚2.270472983519236 0loss/FBt2/assignment_loss‚0.3266948909982703 0loss/SRqqt4/detection_loss‚1.047084677572313 /loss/FBt3/detection_loss‚3.4229945242404938 0loss/FRt1/assignment_loss‚3.1937132545237272 0loss/FRt3/assignment_loss‚3.4488012958795595 %loss/total_loss‚44.91774036074668 % _timestamp‚1.7884028592016397e+09 1loss/SRqqt2/assignment_loss‚2.208586431349869 .loss/FRt2/detection_loss‚3.087362589744421 .loss/FBt1/detection_loss‚2.492024600505829 trainer/global_step‚9549 0loss/FBt1/assignment_loss‚0.1138334475224724 0loss/FBt4/assignment_loss‚0.1806651572630876 2loss/SRqqt1/assignment_loss‚1.2211507517923512 .loss/FRt4/detection_loss‚1.735594261374514 1loss/SRqqt1/detection_loss‚1.7818590219853059 /loss/FBt4/detection_loss‚0.7130396899154198 _runtime‚7483.838384267 _step‚784 0loss/FBt3/assignment_loss‚0.4381576017351563 .loss/FRt1/detection_loss‚2.105287773710377 /loss/FRt3/detection_loss‚2.5882392265857797 1loss/SRqqt2/detection_loss‚2.7088832722769842 epoch‚7 0loss/FRt2/assignment_loss‚3.9180883719370914 0loss/FRt4/assignment_loss‚1.6473942833431696 2loss/SRqqt3/assignment_loss‚1.2381271581080826 0loss/SRqqt4/assignment_loss‚1.299640749711872;ô [‹$Õ %loss/total_loss‚44.91774036074668 % _timestamp‚1.7884028592016397e+09 1loss/SRqqt2/assignment_loss‚2.208586431349869 .loss/FRt2/detection_loss‚3.087362589744421 .loss/FBt1/detection_loss‚2.492024600505829 trainer/global_step‚9549 0loss/FBt1/assignment_loss‚0.1138334475224724 0loss/FBt4/assignment_loss‚0.1806651572630876 2loss/SRqqt1/assignment_loss‚1.2211507517923512 .loss/FRt4/detection_loss‚1.735594261374514 1loss/SRqqt1/detection_loss‚1.7818590219853059 /loss/FBt4/detection_loss‚0.7130396899154198 _runtime‚7483.838384267 _step‚784 0loss/FBt3/assignment_loss‚0.4381576017351563 .loss/FRt1/detection_loss‚2.105287773710377 /loss/FRt3/detection_loss‚2.5882392265857797 1loss/SRqqt2/detection_loss‚2.7088832722769842 epoch‚7 0loss/FRt2/assignment_loss‚3.9180883719370914 0loss/FRt4/assignment_loss‚1.6473942833431696 2loss/SRqqt3/assignment_loss‚1.2381271581080826 0loss/SRqqt4/assignment_loss‚1.29964074971187 1loss/SRqqt3/detection_loss‚1.7300443451512943 .loss/FBt2/detection_loss‚2.270472983519236 0loss/FBt2/assignment_loss‚0.3266948909982703 0loss/SRqqt4/detection_loss‚1.047084677572313 /loss/FBt3/detection_loss‚3.4229945242404938 0loss/FRt1/assignment_loss‚3.1937132545237272 0loss/FRt3/assignment_loss‚3.4488012958795595Jt·Œ$‚(:’ ¬¹ãÔ™êù­ gpu.0.gpu‚32.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚126.905' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.380999999999997 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…l£A|$‚(:ñ ¬¹ãÔ–¡þ¯ disk.drbd2266.out‚0 disk.drbd2265.out‚0* disk./.usagePercent‚15.670984329307775 disk.nvme0c0n1.in‚ 399.828125 disk.dm-1.in‚ 0.06640625 disk.drbd2266.in‚0# disk.dm-0.out‚35193.53271484375$ memory_percent‚96.01306915283203 disk.sda3.out‚ 8021.19140625 disk.sda.out‚ 8021.19140625 disk.drbd1716.in‚0 disk.rbd0.in‚0% proc.memory.availableMB‚ 653.21875! proc.memory.rssMB‚ 2685.171875 network.sent‚ 2.3531142e+07 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚0 cpu‚25.345145476202728 proc.cpu.threads‚17 network.recv‚ 2.693805e+06 disk.dm-1.out‚ 8021.19140625 disk.drbd2080.in‚0 disk.nvme0n1.in‚ 399.828125( disk.nvme0c0n1.out‚35193.53271484375 disk.sda2.in‚0 disk.sda2.out‚0% disk./.usageGB‚233.50899124145508& disk.nvme0n1.out‚35193.53271484375 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 399.828125* proc.memory.percent‚16.388988494873047 disk.sda3.in‚ 0.06640625 disk.rbd1.out‚ 18.99609375 disk.rbd1.in‚0 disk.sda.in‚ 0.06640625Ò~¢ÔŽ$‚(:’ »¹ãÔü冮 gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚134.841' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.968200000000003 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù©Ç±n$‚(:ã »¹ãÔÞó¼° disk.drbd2265.in‚0 network.sent‚ 2.3640102e+07 disk.sda1.out‚0 disk.sda.in‚ 0.06640625! proc.memory.rssMB‚ 2685.171875 proc.cpu.threads‚17 cpu‚25.344586914075187 disk.drbd2266.in‚0 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd1.out‚ 19.05859375 disk.rbd0.out‚0 disk.sda3.out‚ 8021.6875 disk.rbd1.in‚0 disk.sda.out‚ 8021.6875 disk.sda2.out‚0 disk.dm-1.in‚ 0.06640625 disk.dm-1.out‚ 8021.6875 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚0) disk./.usagePercent‚15.67098944945846 disk.drbd2266.out‚0" disk.dm-0.out‚35194.4365234375 disk.sda1.in‚0 disk.sda3.in‚ 0.06640625' disk.nvme0c0n1.out‚35194.4365234375 disk.nvme0n1.in‚ 399.828125* proc.memory.percent‚16.388988494873047$ disk./.usageGB‚233.5090675354004 disk.dm-0.in‚ 399.828125 disk.nvme0c0n1.in‚ 399.828125% disk.nvme0n1.out‚35194.4365234375 network.recv‚ 2.700562e+06 disk.drbd2265.out‚0$ memory_percent‚96.01399898529053( proc.memory.availableMB‚ 653.06640625¯,ÀX$‚(:… ʹãÔÓýï­ gpu.0.gpu‚42.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚123.51' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.702 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0E…–Ó†‘$‚(:û ʹãÔÑÅí¯! proc.memory.rssMB‚ 2685.171875 disk.sda.in‚ 0.06640625 disk.sda.out‚ 8022.109375 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.sda2.in‚0 disk.drbd2266.in‚0# disk.dm-0.out‚35195.07177734375 disk.rbd0.out‚0 disk.sda3.out‚ 8022.109375( disk.nvme0c0n1.out‚35195.07177734375( proc.memory.availableMB‚ 652.80859375* disk./.usagePercent‚15.670985865352982 disk.dm-1.out‚ 8022.109375 disk.rbd0.in‚ 0.0546875" disk.nvme0c0n1.in‚ 400.02734375# memory_percent‚96.0155725479126* proc.memory.percent‚16.388988494873047 cpu‚25.343972071670752 disk.dm-1.in‚ 0.06640625 disk.dm-0.in‚ 400.02734375 disk.nvme0n1.in‚ 400.02734375 network.sent‚ 2.3645436e+07% disk./.usageGB‚233.50901412963867 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.06640625& disk.nvme0n1.out‚35195.07177734375 proc.cpu.threads‚17 network.recv‚ 2.703995e+06 disk.sda2.out‚0 disk.rbd1.out‚ 19.05859375 disk.drbd2266.out‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0¯L2’$- _wandbruntime‚7518  _runtime‚75189E0„É“$à ' lr-AdamW/pg1‚2.6623144624497558e-05 ' lr-AdamW/pg2‚2.6623144624497558e-05 trainer/global_step‚9599 % _timestamp‚1.7884028941098015e+09 _runtime‚7518.746671912 _step‚785Òí½“Î”$È % _timestamp‚1.7884028941098015e+09 _runtime‚7518.746671912 _step‚785 ' lr-AdamW/pg1‚2.6623144624497558e-05 ' lr-AdamW/pg2‚2.6623144624497558e-05 trainer/global_step‚9599‘ân&›2•$- _wandbruntime‚7518  _runtime‚7518…ˆ¶–$æO KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13764531990016 _step‚786 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚283648527638016 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 193867248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4>à²G6&757245257 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚270118093603840 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 193887286 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚544 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13764740477440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚286220688405504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13491433901738 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127990453 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13530247031296 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126259157 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 193867248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚299985196455424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚270450446457514 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚7518.747538286 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚270118111606784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚270118111606784 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13530416031232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73693906 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54296549 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13491268780334 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13530416031232 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127990999 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚299985428882944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚9599 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚283648340635136 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54297093 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 $ _timestamp‚1.788402894110534e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚286220664465408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73693904 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚283648340635136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126259159 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚256958995516416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚256959012555776 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 320146445V¾ßñ'—$ëO N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 320146445 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13764531990016 _step‚786 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚283648527638016 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 193867248 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚270118093603840 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 193887286 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚544 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13764740477440 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚286220688405504 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13491433901738 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 127990453 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13530247031296 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126259157 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 193867248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚299985196455424 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚270450446457514 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚7518.747538286 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚270118111606784 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚270118111606784 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13530416031232 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚73693906 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54296549 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13491268780334 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13530416031232 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 127990999 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚299985428882944 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 trainer/global_step‚9599 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚283648340635136 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54297093 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 $ _timestamp‚1.788402894110534e+09 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚286220664465408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚73693904 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚283648340635136 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126259159 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚256958995516416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚256959012555776 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360’°‘ ÷2˜$- _wandbruntime‚7519  _runtime‚7519\ _ãö&™$ðM ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13765943652352 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 193907099 % _timestamp‚1.7884028947757967e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 126271816 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚73701281 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚537 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 193907099 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚283677190154240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚286249601353728 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13766158417920 _runtime‚ 7519.41312894 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚270145362968576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 320178917 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚270145362968576 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 193887035 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚256984938698752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚283677190154240 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13492823066490 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚787 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54302317 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126271818 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚270145380971520 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128003063 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320178917 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13531809182720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚300015521065984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚270145380971520 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 128003600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13531809182720 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13531640169472 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13531640169472 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13492657944878 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 320158851 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256984955738112 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 193907099 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚283677190154240 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚286249601353728 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13766158417920 _runtime‚ 7519.41312894 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚270145362968576 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 320178917 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚270145362968576 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 193887035 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚256984938698752 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚283677190154240 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13492823066490 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _step‚787 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚54302317 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚535 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126271818 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚270145380971520 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128003063 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 320178917 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13531809182720 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚300015521065984 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚270145380971520 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 128003600 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13531809182720 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13531640169472 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13531640169472 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13492657944878 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 320158851 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚256984955738112 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0“Ä擽2›$- _wandbruntime‚7519  _runtime‚75191:”Zœ$Ô /loss/FRt3/detection_loss‚2.5186680860462642 % _timestamp‚1.7884028947774258e+09 1loss/SRqqt1/assignment_loss‚1.532968303331962 .loss/FBt2/detection_loss‚2.908993152471689 epoch‚7 1loss/FBt1/assignment_loss‚0.37464977504466407 1loss/FBt4/assignment_loss‚0.23832871449014764 trainer/global_step‚9599 /loss/FRt2/assignment_loss‚4.377485448663885 2loss/SRqqt2/assignment_loss‚1.6168019317445301 1loss/SRqqt2/detection_loss‚3.0357185247398557 /loss/FBt4/detection_loss‚0.6471540423401867 %loss/total_loss‚44.92318369079373 1loss/FBt2/assignment_loss‚0.24415172564851856 /loss/FRt2/detection_loss‚3.5840920453721825 1loss/SRqqt1/detection_loss‚1.7546467719456325 _step‚788 .loss/FRt4/detection_loss‚1.873460438535657 .loss/FBt3/detection_loss‚2.518944350155917 0loss/FRt4/assignment_loss‚1.5404945279276647 2loss/SRqqt3/assignment_loss‚1.3044480667449534 2loss/SRqqt4/assignment_loss‚1.1544393209474428 0loss/FBt3/assignment_loss‚0.3842044782569267 _runtime‚7519.414138404 /loss/FRt3/assignment_loss‚2.107952844528925 1loss/SRqqt3/detection_loss‚1.7921453341841698 1loss/SRqqt4/detection_loss‚1.2650942071207931 /loss/FBt1/detection_loss‚2.6091762532790503 /loss/FRt1/assignment_loss‚3.399134963208979 .loss/FRt1/detection_loss‚2.1400303840637315GAp_$Ù .loss/FRt4/detection_loss‚1.873460438535657 .loss/FBt3/detection_loss‚2.518944350155917 0loss/FRt4/assignment_loss‚1.5404945279276647 2loss/SRqqt3/assignment_loss‚1.3044480667449534 2loss/SRqqt4/assignment_loss‚1.1544393209474428 0loss/FBt3/assignment_loss‚0.3842044782569267 _runtime‚7519.414138404 /loss/FRt3/assignment_loss‚2.107952844528925 1loss/SRqqt3/detection_loss‚1.7921453341841698 1loss/SRqqt4/detection_loss‚1.2650942071207931 /loss/FBt1/detection_loss‚2.6091762532790503 /loss/FRt1/assignment_loss‚3.399134963208979 .loss/FRt1/detection_loss‚2.140030384063731 /loss/FRt3/detection_loss‚2.5186680860462642 % _timestamp‚1.7884028947774258e+09 1loss/SRqqt1/assignment_loss‚1.532968303331962 .loss/FBt2/detection_loss‚2.908993152471689 epoch‚7 1loss/FBt1/assignment_loss‚0.37464977504466407 1loss/FBt4/assignment_loss‚0.23832871449014764 trainer/global_step‚9599 /loss/FRt2/assignment_loss‚4.377485448663885 2loss/SRqqt2/assignment_loss‚1.6168019317445301 1loss/SRqqt2/detection_loss‚3.0357185247398557 /loss/FBt4/detection_loss‚0.6471540423401867 %loss/total_loss‚44.92318369079373 1loss/FBt2/assignment_loss‚0.24415172564851856 /loss/FRt2/detection_loss‚3.5840920453721825 1loss/SRqqt1/detection_loss‚1.7546467719456325 _step‚788”XhÉ‘ž$‚(:† Ù¹ãÔŸÂœ® gpu.0.gpu‚32.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚35.0 gpu.0.powerWatts‚145.595' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.119 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¤z輘Ÿ$‚(:  Ù¹ãÔ§›ê°* disk./.usagePercent‚15.670985865352982 disk.nvme0n1.in‚ 400.02734375 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.sda.in‚ 0.06640625 disk.sda.out‚ 8022.61328125 disk.drbd2080.in‚0 disk.dm-0.in‚ 400.02734375 disk.sda1.out‚0% disk./.usageGB‚233.50901412963867" disk.nvme0c0n1.in‚ 400.02734375( disk.nvme0c0n1.out‚35195.86181640625 disk.sda2.in‚0 disk.dm-1.out‚ 8022.61328125 disk.drbd2265.out‚0 disk.rbd0.in‚ 0.0546875 disk.sda3.in‚ 0.06640625 network.recv‚ 2.708149e+06 disk.rbd1.out‚ 19.2705078125# disk.dm-0.out‚35195.86181640625 cpu‚25.343381553726466 disk.sda2.out‚0 disk.dm-1.in‚ 0.06640625 disk.rbd0.out‚ 0.00390625 disk.sda1.in‚0( proc.memory.availableMB‚ 652.99609375 disk.rbd1.in‚0& disk.nvme0n1.out‚35195.86181640625 disk.drbd1716.out‚0! proc.memory.rssMB‚ 2685.171875 disk.drbd1716.in‚0 disk.sda3.out‚ 8022.61328125 network.sent‚ 2.3651398e+07 disk.drbd2266.in‚0$ memory_percent‚96.01442813873291* proc.memory.percent‚16.388988494873047 proc.cpu.threads‚17Ù70“ $‚(:’ è¹ãÔä‹™® gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚130.927' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.185399999999998 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ä€^“¡$‚(:ˆ  è¹ãÔ¹š°% disk./.usageGB‚233.50897598266602 disk.drbd2266.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.06640625( disk.nvme0c0n1.out‚35196.12353515625 cpu‚25.342804577053965! proc.memory.rssMB‚ 2685.171875* proc.memory.percent‚16.388988494873047& disk.nvme0n1.out‚35196.12353515625 disk.drbd1716.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚ 0.0546875 disk.sda1.in‚0 disk.sda3.out‚ 8023.43359375 disk.nvme0n1.in‚ 400.02734375 network.recv‚ 2.713826e+06 disk.dm-1.in‚ 0.06640625 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚ 0.0703125 disk.rbd1.in‚0 disk.sda2.out‚0 proc.cpu.threads‚17 network.sent‚ 2.3759962e+07" disk.nvme0c0n1.in‚ 400.02734375% proc.memory.availableMB‚ 652.53125) disk./.usagePercent‚15.67098330527764 disk.dm-0.in‚ 400.02734375 disk.rbd1.out‚ 19.2705078125 disk.sda.out‚ 8023.43359375 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.sda.in‚ 0.06640625$ memory_percent‚96.01726531982422 disk.dm-1.out‚ 8023.43359375 disk.drbd2266.in‚0# disk.dm-0.out‚35196.12353515625 }è_2¢$- _wandbruntime‚7554  _runtime‚7554ñlWÉ£$à ' lr-AdamW/pg1‚2.5609305043983825e-05 ' lr-AdamW/pg2‚2.5609305043983825e-05 trainer/global_step‚9649 % _timestamp‚1.7884029296205893e+09 _runtime‚7554.257454062 _step‚789Á8q+Τ$È % _timestamp‚1.7884029296205893e+09 _runtime‚7554.257454062 _step‚789 ' lr-AdamW/pg1‚2.5609305043983825e-05 ' lr-AdamW/pg2‚2.5609305043983825e-05 trainer/global_step‚9649•[¼‚E2¥$- _wandbruntime‚7554  _runtime‚75541ÕI²í'¦$çO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚54568555 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 128630765 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 194879988 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 126892109 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚287666335816704 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚271481561840640 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13600087353856 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚258256154633216 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚271481579843584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 194859950 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13560905720146 trainer/global_step‚9649 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13599918353920 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚285081480194560 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126892107 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚271817077392722 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚271816895231958 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚258256171672576 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126892107 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚271481561840640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13560740598742 % _timestamp‚1.7884029296212952e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚301501735428096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚285081480194560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚546 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126892109 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54568011 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚301501967855616 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _step‚790 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13835632038912 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13600087353856 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚287666311876608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13835423551488 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚285081480194560 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 126892107 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚271817077392722 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚271816895231958 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚258256171672576 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 126892107 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚271481561840640 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13560740598742 % _timestamp‚1.7884029296212952e+09 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚301501735428096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚285081480194560 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚546 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 126892109 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚54568011 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚301501967855616 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 _step‚790 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚13835632038912 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13600087353856 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚287666311876608 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13835423551488 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚271481579843584 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 194859950 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13560905720146 trainer/global_step‚9649–ÙHÎ;2¨$- _wandbruntime‚7554  _runtime‚7554Å@öù÷&©$ñM % _timestamp‚1.7884029302961967e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13601311492096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54573403 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚258282097815552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13601480505344 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚301532329793024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7554.933489922 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 321784503 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13562294884898 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚287695224824832 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚285110329713664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74070131 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚271508849208320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 321804569 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _step‚791 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 194899801 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74070133 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚271844409739810 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126904768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 321784503 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126904768 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13837081028096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13836874651136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚271508831205376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚285110142697472 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128643534 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13601311492096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0+ Äü&ª$öM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚285110329713664 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74070131 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚271508849208320 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚524 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 321804569 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _step‚791 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 194899801 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74070133 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚271844409739810 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 126904768 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 321784503 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 126904768 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13837081028096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13836874651136 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚271508831205376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚285110142697472 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 128643534 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13601311492096 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 % _timestamp‚1.7884029302961967e+09 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13601311492096 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚54573403 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚258282097815552 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13601480505344 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚301532329793024 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚7554.933489922 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 321784503 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13562294884898 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚287695224824832 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416—­?Wq2«$- _wandbruntime‚7554  _runtime‚7554h¯u>Z¬$Ô 0loss/SRqqt4/detection_loss‚1.396252825781703 /loss/FBt4/detection_loss‚0.7248071349336939 0loss/FRt2/assignment_loss‚3.0917490615564236 0loss/FRt4/assignment_loss‚1.8885145872831344 2loss/SRqqt2/assignment_loss‚2.2194841581006206 1loss/FBt2/assignment_loss‚0.43569585123245463 0loss/FBt4/assignment_loss‚0.1544124389391728 1loss/SRqqt2/detection_loss‚2.2035108737407194 2loss/SRqqt4/assignment_loss‚0.8729326295852661 .loss/FRt2/detection_loss‚2.766336523434695 _step‚792 .loss/FBt3/detection_loss‚2.342016279697418 0loss/FBt1/assignment_loss‚0.1881165846741727 trainer/global_step‚9649 % _timestamp‚1.7884029302978985e+09 2loss/SRqqt1/assignment_loss‚1.6344543982435156 .loss/FRt3/detection_loss‚2.552898595207616 /loss/FBt2/detection_loss‚2.3810938734274645 %loss/total_loss‚44.32783810955824 0loss/FRt1/assignment_loss‚3.1311108203870908 0loss/FRt3/assignment_loss‚2.8584083663789848 .loss/FRt1/detection_loss‚2.068906161401953 1loss/SRqqt1/detection_loss‚1.7568068676800639 1loss/SRqqt3/detection_loss‚1.9898140202177332 epoch‚7 .loss/FRt4/detection_loss‚2.206056628935039 /loss/FBt1/detection_loss‚3.6975324804132637 _runtime‚7554.934640462 2loss/SRqqt3/assignment_loss‚1.3376453920723277 0loss/FBt3/assignment_loss‚0.4292815562337182ó¶4¾_­$Ù .loss/FRt3/detection_loss‚2.552898595207616 /loss/FBt2/detection_loss‚2.3810938734274645 %loss/total_loss‚44.32783810955824 0loss/FRt1/assignment_loss‚3.1311108203870908 0loss/FRt3/assignment_loss‚2.8584083663789848 .loss/FRt1/detection_loss‚2.068906161401953 1loss/SRqqt1/detection_loss‚1.7568068676800639 1loss/SRqqt3/detection_loss‚1.9898140202177332 epoch‚7 .loss/FRt4/detection_loss‚2.206056628935039 /loss/FBt1/detection_loss‚3.6975324804132637 _runtime‚7554.934640462 2loss/SRqqt3/assignment_loss‚1.3376453920723277 0loss/FBt3/assignment_loss‚0.4292815562337182 0loss/SRqqt4/detection_loss‚1.396252825781703 /loss/FBt4/detection_loss‚0.7248071349336939 0loss/FRt2/assignment_loss‚3.0917490615564236 0loss/FRt4/assignment_loss‚1.8885145872831344 2loss/SRqqt2/assignment_loss‚2.2194841581006206 1loss/FBt2/assignment_loss‚0.43569585123245463 0loss/FBt4/assignment_loss‚0.1544124389391728 1loss/SRqqt2/detection_loss‚2.2035108737407194 2loss/SRqqt4/assignment_loss‚0.8729326295852661 .loss/FRt2/detection_loss‚2.766336523434695 _step‚792 .loss/FBt3/detection_loss‚2.342016279697418 0loss/FBt1/assignment_loss‚0.1881165846741727 trainer/global_step‚9649 % _timestamp‚1.7884029302978985e+09 2loss/SRqqt1/assignment_loss‚1.6344543982435156˜ƒ˜°®$‚(:’ ÷¹ãÔ¯ gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚148.312' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.662399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¸7?…¯$‚(:„  ÷¹ãÔÏãê° disk.dm-0.in‚ 400.02734375# disk.dm-0.out‚35196.61865234375& disk.nvme0n1.out‚35196.61865234375 disk.drbd2080.in‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0( proc.memory.availableMB‚ 652.76953125 network.recv‚ 2.718479e+06* disk./.usagePercent‚15.670995849646815 disk.drbd2080.out‚0 disk.rbd0.in‚ 0.0546875 disk.rbd1.out‚ 19.3330078125 disk.nvme0n1.in‚ 400.02734375 disk.drbd2266.out‚0 disk.sda3.in‚ 0.06640625 disk.dm-1.in‚ 0.06640625# memory_percent‚96.0158109664917* proc.memory.percent‚16.388988494873047 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 400.02734375 disk.sda.out‚ 8024.078125 disk.sda2.in‚0 cpu‚25.34222113498658 network.sent‚ 2.3766231e+07( disk.nvme0c0n1.out‚35196.61865234375 disk.drbd1716.out‚0 disk.dm-1.out‚ 8024.078125 disk.drbd2266.in‚0 disk.rbd0.out‚ 0.0703125 disk.sda3.out‚ 8024.078125 disk.rbd1.in‚0 disk.sda.in‚ 0.06640625 disk.sda2.out‚0! proc.memory.rssMB‚ 2685.171875 proc.cpu.threads‚17% disk./.usageGB‚233.509162902832030u[C°$‚(:’ †ºãÔ§³ä¯ gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚150.486' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.097199999999997 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0á÷ÐÕŽ±$‚(:ƒ  †ºãÔÁÌ»° disk.dm-1.in‚ 0.06640625 disk.dm-1.out‚ 8024.64453125 disk.drbd2265.in‚0 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 400.02734375 disk.sda2.out‚0) proc.memory.percent‚16.44418239593506 network.recv‚ 2.722979e+06 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 8024.64453125 disk.sda.in‚ 0.06640625 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.rbd0.out‚ 0.1484375 network.sent‚ 2.3873359e+07% disk./.usageGB‚233.50910568237305 disk.drbd2266.in‚0 disk.dm-0.in‚ 400.02734375 cpu‚25.34398249676827" proc.memory.availableMB‚643.75# proc.memory.rssMB‚ 2694.21484375 disk.rbd1.in‚0 disk.rbd1.out‚ 19.3330078125% disk.nvme0n1.out‚35196.9912109375" disk.dm-0.out‚35196.9912109375 disk.sda3.in‚ 0.06640625 disk.drbd1716.in‚0 disk.sda.out‚ 8024.64453125* disk./.usagePercent‚15.670992009533801 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 400.02734375$ memory_percent‚96.07086181640625 proc.cpu.threads‚17 disk.drbd2080.in‚0 disk.rbd0.in‚ 0.0546875' disk.nvme0c0n1.out‚35196.9912109375›ÒÚH’²$‚(:‡ •ºãÔ¯ïñ® gpu.0.gpu‚10.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚122.154' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CQò·³$‚(:„  •ºãÔâɸ±* disk./.usagePercent‚15.670990473488597% disk./.usageGB‚233.50908279418945 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 8024.984375* proc.memory.percent‚16.445374488830566 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.in‚ 0.06640625 disk.rbd1.out‚ 19.458984375" disk.nvme0c0n1.in‚ 400.02734375 disk.nvme0n1.in‚ 400.02734375% disk.nvme0n1.out‚35197.9697265625 disk.sda.in‚ 0.06640625 disk.drbd2266.in‚0 network.recv‚ 2.727231e+06 disk.dm-1.in‚ 0.06640625 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚ 0.0546875$ memory_percent‚96.06983661651611 cpu‚25.353606781030624 disk.sda1.in‚0 disk.sda.out‚ 8024.984375 disk.drbd1716.out‚0 disk.rbd0.out‚ 0.1484375( proc.memory.availableMB‚ 643.91796875 network.sent‚ 2.3879498e+07 disk.dm-0.in‚ 400.02734375 disk.sda3.out‚ 8024.984375 disk.drbd2265.out‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0" disk.dm-0.out‚35197.9697265625' disk.nvme0c0n1.out‚35197.9697265625 disk.sda2.in‚0 disk.sda2.out‚0# proc.memory.rssMB‚ 2694.41015625#²Î‚´$‚(:… ¤ºãÔÎáø­ gpu.0.gpu‚9.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚122.395' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.479 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z|Œµ$‚(:  ¤ºãÔ ¸“± network.recv‚ 2.731402e+06" disk.nvme0c0n1.in‚ 400.02734375 disk.sda.in‚ 0.06640625( proc.memory.availableMB‚ 643.73046875 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 8027.890625 disk.drbd2265.out‚0 disk.rbd0.in‚ 0.0546875 disk.rbd0.out‚ 0.21484375 disk.sda1.out‚0 disk.nvme0n1.in‚ 400.02734375 disk.sda2.in‚0 disk.drbd2266.in‚0# proc.memory.rssMB‚ 2694.41015625! disk.dm-0.out‚35198.580078125$ disk.nvme0n1.out‚35198.580078125 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2080.in‚0* disk./.usagePercent‚15.670943368102296% disk./.usageGB‚233.50838088989258 disk.sda3.in‚ 0.06640625 disk.sda3.out‚ 8027.890625 disk.drbd2265.in‚0# memory_percent‚96.0709810256958* proc.memory.percent‚16.445374488830566 cpu‚25.363300426080436 network.sent‚ 2.3885528e+07 disk.rbd1.out‚ 19.458984375& disk.nvme0c0n1.out‚35198.580078125 disk.sda.out‚ 8027.890625 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 400.02734375 disk.sda1.in‚0ä¿ï2¶$- _wandbruntime‚7619  _runtime‚7619[Ÿ»Æ·$ÍT FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚217 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 4541568000 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 323389806 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚259730040995840 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚286682424881152 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2460 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚284 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚26 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 $ _timestamp‚1.788402994491326e+09 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚273033169336320 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚13609784833606 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 145311232 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚286682279569920 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚273339949755038 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚13649128236544 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 5041553408 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 129489207 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚74550927 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 410712064 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 140964952 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 trainer/global_step‚9671 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚303177203481088 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚416ï]°X*¸$ÒT K=DeviceStatsMonitor.on_validation_batch_start/active.all.freed‚ 323389806 JDDeviceStatsMonitor.on_validation_batch_start/oversize_segments.freed‚0 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚259730040995840 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚286682424881152 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2460 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚284 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚26 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 $ _timestamp‚1.788402994491326e+09 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 eQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.allocated‚273033169336320 E?DeviceStatsMonitor.on_validation_batch_start/num_oom_rejections‚0 NHDeviceStatsMonitor.on_validation_batch_start/oversize_segments.allocated‚0 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚13609784833606 KEDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 145311232 ZFDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.freed‚286682279569920 ^JDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.allocated‚273339949755038 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚13649128236544 SDDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.peak‚ 5041553408 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 129489207 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.allocated‚74550927 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 \ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_validation_batch_start/active.all.current‚15979 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 410712064 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 VHDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.current‚ 140964952 KEDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.freed‚0 trainer/global_step‚9671 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚303177203481088 ]NDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_validation_batch_start/segment.all.allocated‚416 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 OGDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.current‚217 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 4541568000 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 KEDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.freed‚0™Uªà^‘¹$‚(:† ³ºãÔ𮯠gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚122.154' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 Ôu ‡º$‚(:ü ³ºãÔÓÿ¼± disk.drbd2080.out‚0 disk.rbd0.out‚ 0.21484375 cpu‚25.372624545365507 disk.nvme0n1.in‚ 400.02734375 disk.sda2.out‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2266.in‚0 disk.drbd2266.out‚0* disk./.usagePercent‚15.671079820118047 disk.dm-0.in‚ 400.02734375# proc.memory.rssMB‚ 2694.41015625 network.recv‚ 2.736139e+06 disk.nvme0n1.out‚ 35204.46875 disk.rbd1.out‚ 19.521484375 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.3891828e+07 disk.sda.out‚ 8031.73828125 disk.dm-1.out‚ 8031.73828125 disk.drbd2265.out‚0 disk.rbd0.in‚ 0.0546875 disk.sda1.in‚0 disk.sda3.out‚ 8031.73828125* proc.memory.percent‚16.445374488830566" disk.nvme0c0n1.out‚ 35204.46875 disk.sda3.in‚ 0.06640625( proc.memory.availableMB‚ 645.94921875" disk.nvme0c0n1.in‚ 400.02734375 disk.sda.in‚ 0.06640625 disk.sda1.out‚0$ memory_percent‚96.05743885040283 disk.dm-0.out‚ 35204.46875% disk./.usageGB‚233.51041412353516 disk.sda2.in‚0 disk.drbd2265.in‚0Ïð½2»$- _wandbruntime‚7620  _runtime‚7620Y0+^)¼$ØR PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 _runtime‚7620.055084968 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 127308288 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚416 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 323423893 MGDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.current‚0 JDDeviceStatsMonitor.on_validation_batch_end/active.large_pool.current‚2 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚985 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 5041553408 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚199 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚289311487964160 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 417333248 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚259747925510144 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚13610256365534 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚217 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 195705027 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 129500971 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 269877760 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚273358198915038 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 140964952 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 5041553408 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 _step‚794 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚13649604680704 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚13649604680704 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 4757245257 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚54943544 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 4912020992 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚289311511904256 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚286701493891584 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1006 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚273358057950086 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 127718866 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2460 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚74557427 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚303197291889664 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚303197156182528 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚273052016519168 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 127718864 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 195689050 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 111767040 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚286701639202816 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 323423893 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 4912020992 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚13649477372416 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 410712064 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 417333248 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 127718866 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 5041553408 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚985 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 5041553408 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 JBDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.peak‚199 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚289311487964160 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 417333248 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚259747925510144 bODeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.allocated‚13610256365534 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚217 ICDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.freed‚0 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 195705027 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 129500971 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 269877760 \HDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.allocated‚273358198915038 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 140964952 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 5041553408 NDDeviceStatsMonitor.on_validation_batch_end/active.small_pool.current‚15977 _step‚794 _LDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.allocated‚13649604680704 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚13649604680704 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 4757245257 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚54943544 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 RCDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.peak‚ 4912020992 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚289311511904256 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UADeviceStatsMonitor.on_validation_batch_end/active_bytes.all.freed‚286701493891584 ZMDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1006 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚273358057950086 XJDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.allocated‚ 127718866 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2460 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚74557427 XIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚303197291889664 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚303197156182528 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚273052016519168 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 127718864 VGDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 195689050 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 111767040 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚286701639202816 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 323423893 O@DeviceStatsMonitor.on_validation_batch_end/active_bytes.all.peak‚ 4912020992 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚13649477372416 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 410712064 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 @9DeviceStatsMonitor.on_validation_batch_end/max_split_size‚-1 ZLDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.current‚ 417333248 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 127718866 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 5041553408 B&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5432099103927612 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_7‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2155013233423233 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1448277235031128 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.48597291111946106 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14227667450904846 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.07073580473661423 *jet/accuracy_1_of_5‚0.9435996413230896 particle/accuracy_7_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5833332538604736 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4815669655799866 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9852967858314514 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.009442497976124287 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9918952584266663 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29268568754196167 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,jet/accuracy_5_of_5‚0.001792114693671465 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_7‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5574713349342346 *jet/accuracy_1_of_1‚0.4896087646484375 particle/accuracy_3_of_4‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9898874163627625 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03236844390630722 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23629449307918549 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4062289595603943 particle/accuracy_6_of_7‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5261833071708679 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5164401531219482 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2658730149269104 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.24232155084609985 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3194444477558136 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4134567975997925 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.611687183380127 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 particle/accuracy_4_of_4‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3958333432674408 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22335030138492584 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5351482033729553 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5468001365661621 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2620629072189331 8"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.474801629781723 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.18103450536727905 jet/accuracy_4_of_7‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.40900230407714844 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07720856368541718 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.030426660552620888 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03568011522293091 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 6 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.309259295463562 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267138838768005 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16153816878795624 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16154570877552032 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1541973501443863 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19360806047916412 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.41928109526634216 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.36733630299568176 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5598214268684387 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18759968876838684 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9742511510848999 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9908296465873718 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9869614839553833 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4811479449272156 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13628432154655457 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_6‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 particle/accuracy_5_of_7‚1 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.816013514995575 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5123833417892456 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13696415722370148 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_6‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2596481144428253 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_5‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18243560194969177 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.23275861144065857 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18478311598300934 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3286990523338318 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 particle/accuracy_5_of_6‚1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10283688455820084 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5225912928581238 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9899340271949768 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4720301926136017 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 +jet/accuracy_3_of_6‚0.13461539149284363 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5840277075767517 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 jet/accuracy_3_of_7‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9974644780158997 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9662356376647949 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1953490823507309 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9889993071556091 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9789473414421082 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11058416962623596 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20266658067703247 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03412069007754326 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5833333134651184 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09799584001302719 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.27500003576278687 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9928423166275024 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.19066697359085083 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.41470175981521606 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4099308252334595 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_6_of_7‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5629011988639832 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740740299224854 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 )loss/val_total_loss‚44.33771330629623 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5510960817337036 particle/accuracy_1_of_2‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_6‚0.9807692170143127 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5726190209388733 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2222222238779068 particle/accuracy_4_of_6‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 +particle/sensitivity‚0.2981812357902527 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 _runtime‚7633.026646133 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.018492067232728004 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0035298035945743322 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 jet/accuracy_2_of_7‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5527852177619934 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18110719323158264 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.44052502512931824 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22247271239757538 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07023809105157852 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.025415224954485893 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 (particle/accuracy‚0.8457702398300171 +particle/specificity‚0.9471103549003601 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4548611640930176 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.6410256028175354 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997404932975769 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.264066219329834 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0022675737272948027 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 epoch‚7 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9884197115898132 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9759185314178467 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.971230149269104 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2590090036392212 *jet/accuracy_1_of_4‚0.9115724563598633 +jet/accuracy_2_of_4‚0.39905813336372375 _step‚795 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3189564645290375 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_2‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.28409090638160706 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5832315683364868 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.34181854128837585 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012983965687453747 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1440182626247406 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3681240677833557 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5126479864120483 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2006908804178238 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.12222222238779068 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5745264291763306 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.358079195022583 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1269841343164444 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13182199001312256 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4862925708293915 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09567900747060776 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.40360358357429504 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4215686619281769 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6486783027648926 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.23484846949577332 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3194444477558136 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9903664588928223 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5583435893058777 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.040697675198316574 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 6validation_average_jet_accuracy‚0.4387410283088684 particle/accuracy_3_of_6‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.057232704013586044 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1702294796705246 ;"Purity/2FRt1SRqqt2FBt/event_purity‚0.022727273404598236 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5142977237701416 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.4736842215061188 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 A&Purity/3FRt4SRqqto˜i •9*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 *jet/accuracy_4_of_4‚0.0023547881282866 ,jet/accuracy_4_of_5‚0.004096262622624636 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5115740895271301 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5833333134651184 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41772499680519104 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5551108121871948 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.49193626642227173 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5199713110923767 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5074074268341064 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5548233985900879 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013571502640843391 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1844136118888855 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1702294796705246 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.580872654914856 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5535714030265808 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4988841116428375 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 jet/accuracy_5_of_6‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15564177930355072 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.48499131202697754 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.10823754966259003 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4978194832801819 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.578891396522522 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9890533685684204 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5549352765083313 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14669188857078552 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5304128527641296 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4120841920375824 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735631346702576 particle/accuracy_4_of_5‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964737296104431 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1485414057970047 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_2_of_4‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 (particle/f_score‚0.37623605132102966 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 % _timestamp‚1.7884030083878503e+09 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.19130347669124603 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.45504388213157654 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5390003323554993 *jet/accuracy_1_of_3‚0.8655908703804016 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5123833417892456 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 6 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.166666641831398 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10902566462755203 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.41559457778930664 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5455908179283142 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26675426959991455 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07988999783992767 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 *jet/accuracy_1_of_2‚0.7551319003105164 particle/accuracy_1_of_1‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2222222238779068 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.35784831643104553 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08886542916297913 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.159311905503273 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08023334294557571 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123456716537476 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25250956416130066 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2690241038799286 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10498550534248352 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9845502972602844 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.41890016198158264 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9942410588264465 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5684915781021118 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2532893419265747 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4934707283973694 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5405021905899048 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.021844293922185898 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.45333331823349 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18876047432422638 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5069507956504822 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0385473258793354 particle/accuracy_2_of_3‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.361249715089798 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5414493680000305 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13162624835968018 ,"Purity/0FRt0SRqqt3FBt/event_purity‚0.125 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16971130669116974 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.02083333395421505 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 jet/accuracy_1_of_7‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6983209252357483 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 +jet/accuracy_2_of_2‚0.11229486018419266 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.539128303527832 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5125226378440857 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 jet/accuracy_6_of_6‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9835734963417053 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3895348906517029 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.43938812613487244 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.013658473268151283 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9823307394981384 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.47764599323272705 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3635416328907013 +jet/accuracy_3_of_5‚0.08680234849452972 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973471164703369 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3684028089046478 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38794130086898804 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4116858243942261 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 trainer/global_step‚9671 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6666666865348816 particle/accuracy_1_of_7‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16473233699798584 particle/accuracy_3_of_5‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1894736886024475 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0µe·ê]FÀ$«¦ 8 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.40900230407714844 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07720856368541718 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9936092495918274 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.030426660552620888 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03568011522293091 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 6 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.309259295463562 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267138838768005 :"Purity/1FRt*SRqqt1FBt/event_purity‚0.16153816878795624 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 8 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.16154570877552032 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1541973501443863 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19360806047916412 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.41928109526634216 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.36733630299568176 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5598214268684387 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 jet/accuracy_7_of_7‚0 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18759968876838684 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9742511510848999 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9908296465873718 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9869614839553833 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4811479449272156 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.13628432154655457 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_1_of_6‚1 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 particle/accuracy_5_of_7‚1 8"Purity/0FRt0SRqqt*FBt/event_purity‚0.816013514995575 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5123833417892456 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.13696415722370148 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444179534912 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ( Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 jet/accuracy_4_of_6‚0 7 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.2596481144428253 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_5_of_5‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18243560194969177 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 ) Purity/3FRt1SRqqt2FBt/FRt_purity‚0.25 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 8 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.23275861144065857 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 :"Purity/*FRt*SRqqt2FBt/event_purity‚0.18478311598300934 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3286990523338318 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 particle/accuracy_1_of_3‚1 particle/accuracy_5_of_6‚1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ) Purity/*FRt2SRqqt2FBt/FRt_purity‚0.25 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.10283688455820084 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5225912928581238 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9899340271949768 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4720301926136017 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 +jet/accuracy_3_of_6‚0.13461539149284363 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5840277075767517 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 jet/accuracy_3_of_7‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9974644780158997 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9662356376647949 7 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.1953490823507309 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 ( Purity/*FRt0SRqqt3FBt/FRt_purity‚0.5 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9889993071556091 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9789473414421082 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11058416962623596 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20266658067703247 :"Purity/2FRt*SRqqt1FBt/event_purity‚0.03412069007754326 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5833333134651184 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09799584001302719 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.27500003576278687 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9928423166275024 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.19066697359085083 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.41470175981521606 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4099308252334595 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/*FRt0SRqqt4FBt/event_purity‚0 jet/accuracy_6_of_7‚0 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5629011988639832 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 :"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.40740740299224854 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 )loss/val_total_loss‚44.33771330629623 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5510960817337036 particle/accuracy_1_of_2‚1 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 *jet/accuracy_1_of_6‚0.9807692170143127 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5726190209388733 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.2222222238779068 particle/accuracy_4_of_6‚1 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 +particle/sensitivity‚0.2981812357902527 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 _runtime‚7633.026646133 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9896175265312195 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.018492067232728004 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0035298035945743322 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 jet/accuracy_2_of_7‚0 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5527852177619934 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/0FRt1SRqqt1FBt/event_purity‚0.18110719323158264 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.44052502512931824 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22247271239757538 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.07023809105157852 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 ,jet/accuracy_3_of_3‚0.025415224954485893 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 (particle/accuracy‚0.8457702398300171 +particle/specificity‚0.9471103549003601 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4548611640930176 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 *jet/accuracy_2_of_6‚0.6410256028175354 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997404932975769 6 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.264066219329834 <"Purity/2FRt1SRqqt*FBt/event_purity‚0.0022675737272948027 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 7 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 epoch‚7 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9884197115898132 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9759185314178467 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 6 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.971230149269104 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.2590090036392212 *jet/accuracy_1_of_4‚0.9115724563598633 +jet/accuracy_2_of_4‚0.39905813336372375 _step‚795 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3189564645290375 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_2_of_2‚1 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 8 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.28409090638160706 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 * Purity/3FRt3SRqqt0FBt/FRt_purity‚0.125 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5832315683364868 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.34181854128837585 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.012983965687453747 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1440182626247406 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.3681240677833557 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5126479864120483 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2006908804178238 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.12222222238779068 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5745264291763306 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 6 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.358079195022583 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1269841343164444 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13182199001312256 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4862925708293915 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.09567900747060776 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 ("Purity/1FRt1SRqqt2FBt/event_purity‚0 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.40360358357429504 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4215686619281769 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3928571343421936 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6486783027648926 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ) Purity/4FRt2SRqqt0FBt/FRt_purity‚0.25 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.23484846949577332 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3194444477558136 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9903664588928223 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5583435893058777 ;"Purity/2FRt*SRqqt2FBt/event_purity‚0.040697675198316574 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 6validation_average_jet_accuracy‚0.4387410283088684 particle/accuracy_3_of_6‚1 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.057232704013586044 7 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.1702294796705246 ;"Purity/2FRt1SRqqt2FBt/event_purity‚0.022727273404598236 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 particle/accuracy_2_of_6‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5142977237701416 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/*FRt*SRqqt3FBt/FRt_purity‚0.4736842215061188 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 particle/accuracy_1_of_5‚1 particle/accuracy_2_of_5‚1 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 *"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.5 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 * Purity/3FRt3SRqqt*FBt/FRt_purity‚0.125 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 *jet/accuracy_4_of_4‚0.0023547881282866 ,jet/accuracy_4_of_5‚0.004096262622624636 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5115740895271301 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 >&Purity/1FRt0SRqqt2FBt/evú³ÚqÕLent_proportion‚0.01512896828353405 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5833333134651184 :"Purity/*FRt*SRqqt1FBt/event_purity‚0.41772499680519104 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5551108121871948 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 :"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.49193626642227173 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5199713110923767 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 ("Purity/*FRt*SRqqt4FBt/event_purity‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.2666666805744171 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5074074268341064 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5548233985900879 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013571502640843391 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.1844136118888855 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 9"Purity/1FRt0SRqqt0FBt/event_purity‚0.1702294796705246 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.580872654914856 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/4FRt2SRqqt*FBt/FRt_purity‚0.25 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5535714030265808 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4988841116428375 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 jet/accuracy_5_of_6‚0 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15564177930355072 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.48499131202697754 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.10823754966259003 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_6_of_6‚1 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 9"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.4978194832801819 8"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.578891396522522 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9890533685684204 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5549352765083313 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14669188857078552 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚0 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5304128527641296 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4120841920375824 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9735631346702576 particle/accuracy_4_of_5‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964737296104431 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1485414057970047 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 particle/accuracy_2_of_4‚1 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 (particle/f_score‚0.37623605132102966 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 % _timestamp‚1.7884030083878503e+09 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.19130347669124603 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.45504388213157654 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5390003323554993 *jet/accuracy_1_of_3‚0.8655908703804016 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5123833417892456 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 6 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.166666641831398 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.10902566462755203 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.41559457778930664 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5455908179283142 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.26675426959991455 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07988999783992767 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 *jet/accuracy_1_of_2‚0.7551319003105164 particle/accuracy_1_of_1‚1 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.2222222238779068 6 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.190476194024086 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.35784831643104553 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08886542916297913 6 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.159311905503273 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08023334294557571 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 8 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.23333333432674408 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 :"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.40123456716537476 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25250956416130066 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2690241038799286 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7272727489471436 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.10498550534248352 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9845502972602844 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.41890016198158264 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9942410588264465 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5684915781021118 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 9"Purity/*FRt0SRqqt2FBt/event_purity‚0.2532893419265747 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4934707283973694 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5405021905899048 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 ("Purity/0FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.6969696879386902 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.021844293922185898 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 7"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.45333331823349 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.18876047432422638 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5069507956504822 9"Purity/1FRt1SRqqt1FBt/event_purity‚0.0385473258793354 particle/accuracy_2_of_3‚1 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 6 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.361249715089798 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.6428571343421936 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5414493680000305 8 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.13162624835968018 ,"Purity/0FRt0SRqqt3FBt/event_purity‚0.125 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.16971130669116974 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 :"Purity/0FRt1SRqqt2FBt/event_purity‚0.02083333395421505 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 jet/accuracy_1_of_7‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6983209252357483 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 +jet/accuracy_2_of_2‚0.11229486018419266 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.539128303527832 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5125226378440857 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 jet/accuracy_6_of_6‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9835734963417053 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3895348906517029 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 +jet/accuracy_2_of_5‚0.43938812613487244 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.013658473268151283 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9823307394981384 :"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.47764599323272705 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3635416328907013 +jet/accuracy_3_of_5‚0.08680234849452972 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.9973471164703369 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3684028089046478 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/*FRt0SRqqt*FBt/event_purity‚0.38794130086898804 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4116858243942261 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 trainer/global_step‚9671 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6666666865348816 particle/accuracy_1_of_7‚1 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16473233699798584 particle/accuracy_3_of_5‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 7 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.1894736886024475 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5432099103927612 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 jet/accuracy_5_of_7‚0 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 *"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.5 + Purity/4FRt*SRqqt*FBt/FRt_purity‚0.1875 7 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.2155013233423233 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1448277235031128 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.48597291111946106 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14227667450904846 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 +jet/accuracy_3_of_4‚0.07073580473661423 *jet/accuracy_1_of_5‚0.9435996413230896 particle/accuracy_7_of_7‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5833332538604736 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4815669655799866 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9852967858314514 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.009442497976124287 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_7‚1 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9918952584266663 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29268568754196167 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,jet/accuracy_5_of_5‚0.001792114693671465 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_4_of_7‚1 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 9"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.5574713349342346 *jet/accuracy_1_of_1‚0.4896087646484375 particle/accuracy_3_of_4‚1 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚0 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9898874163627625 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03236844390630722 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23629449307918549 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4062289595603943 particle/accuracy_6_of_7‚1 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5261833071708679 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5164401531219482 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2658730149269104 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 +jet/accuracy_2_of_3‚0.24232155084609985 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3194444477558136 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4134567975997925 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 , Purity/1FRt0SRqqt3FBt/FRt_purity‚0.53125 8"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.611687183380127 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 particle/accuracy_4_of_4‚1 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.09210526198148727 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3958333432674408 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22335030138492584 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5351482033729553 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05263157933950424 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 ("Purity/0FRt0SRqqt4FBt/event_purity‚0 + Purity/4FRt*SRqqt0FBt/FRt_purity‚0.1875 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5468001365661621 9"Purity/0FRt*SRqqt2FBt/event_purity‚0.2620629072189331 8"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.474801629781723 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.18103450536727905 jet/accuracy_4_of_7‚0 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875›̖߶Á$‚(:’ ºãÔãÉ•® gpu.0.gpu‚11.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚32.0 gpu.0.powerWatts‚101.377' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚20.275399999999998 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÂiÖëˆÂ$‚(:ý ºãÔº˜®° network.recv‚ 2.741735e+06 disk.dm-1.in‚ 0.06640625 disk.drbd2265.out‚0* proc.memory.percent‚17.026901245117188 proc.cpu.threads‚17* disk./.usagePercent‚15.671107468931744 disk.sda1.out‚0 proc.memory.rssMB‚ 2789.6875 disk.sda1.in‚0 disk.sda.in‚ 0.06640625 disk.sda2.out‚0 disk.drbd2080.in‚0$ memory_percent‚96.63608074188232 disk.nvme0n1.in‚ 400.02734375 disk.sda.out‚ 8032.19921875 disk.dm-1.out‚ 8032.19921875 disk.rbd0.out‚ 0.26220703125 disk.dm-0.out‚ 35204.765625( proc.memory.availableMB‚ 551.14453125" disk.nvme0c0n1.in‚ 400.02734375# disk.nvme0c0n1.out‚ 35204.765625! disk.nvme0n1.out‚ 35204.765625 disk.drbd2266.out‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 19.521484375 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.rbd0.in‚ 0.0546875 disk.dm-0.in‚ 400.02734375 cpu‚25.38074731058073% disk./.usageGB‚233.51082611083984 disk.sda3.out‚ 8032.19921875 disk.drbd1716.in‚0 disk.drbd2265.in‚0 network.sent‚ 2.3988967e+07 disk.sda3.in‚ 0.06640625 disk.drbd2080.out‚0Ø ÓÓÃ$‚(:’ ѺãÔÇ駮 gpu.0.gpu‚24.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚136.974' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.394799999999996 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0T†ÞÄ$‚(:„  ѺãÔ†³‰²* disk./.usagePercent‚15.671107468931744 disk.sda1.in‚0 disk.sda3.in‚ 0.06640625 disk.sda.in‚ 0.06640625 network.sent‚ 2.4123381e+07 disk.drbd2266.out‚0 disk.drbd2080.out‚0 network.recv‚ 2.74801e+06 disk.sda1.out‚0 disk.rbd1.out‚ 19.6474609375& disk.nvme0n1.out‚35330.64697265625 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 400.02734375 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.rbd0.in‚ 0.0546875 disk.dm-0.in‚ 400.02734375) proc.memory.percent‚16.44587516784668 proc.cpu.threads‚17% disk./.usageGB‚233.51082611083984 disk.sda3.out‚ 8032.84375( disk.nvme0c0n1.out‚35330.64697265625# disk.dm-0.out‚35330.64697265625" proc.memory.rssMB‚ 2694.4921875 disk.sda.out‚ 8032.84375 disk.dm-1.in‚ 0.06640625 disk.dm-1.out‚ 8032.84375' proc.memory.availableMB‚ 645.9609375 disk.rbd1.in‚0 disk.drbd2266.in‚0 disk.rbd0.out‚ 0.26611328125 cpu‚25.379958830033818" disk.nvme0c0n1.in‚ 400.02734375 disk.drbd1716.in‚0# memory_percent‚96.0573673248291ÎjÞ›2Å$- _wandbruntime‚7654  _runtime‚7654¨ÔÏ!ÉÆ$à ' lr-AdamW/pg1‚2.4612529966709542e-05 ' lr-AdamW/pg2‚2.4612529966709542e-05 trainer/global_step‚9699 % _timestamp‚1.7884030301488745e+09 _runtime‚7654.785724137 _step‚796Ðb^ÎÇ$È ' lr-AdamW/pg1‚2.4612529966709542e-05 ' lr-AdamW/pg2‚2.4612529966709542e-05 trainer/global_step‚9699 % _timestamp‚1.7884030301488745e+09 _runtime‚7654.785724137 _step‚796œLž’å2È$- _wandbruntime‚7654  _runtime‚7654Ù Ž%™$É$çO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55152726 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128199915 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 324520450 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13652293845100 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚9699 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚274333240354544 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 129993832 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚290347007858688 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚274333422515308 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13691602227200 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 196340575 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚290346983918592 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚287725326012416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚304275713419776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74841106 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55152197 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 324540490 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 196320537 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 % _timestamp‚1.7884030301495543e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚274033536782336 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 196320537 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13928497073664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚287725139009536 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128199913 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129993301 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13691771227136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚274033554785280 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13691602227200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128199913 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 196340575 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚287725139009536 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74841104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚260681111630848 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚290347007858688 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚274333422515308 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13691602227200 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 196340575 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚290346983918592 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚287725326012416 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚304275713419776 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚74841106 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55152197 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 324540490 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 196320537 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 % _timestamp‚1.7884030301495543e+09 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚274033536782336 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 196320537 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚13928497073664 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚287725139009536 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 128199913 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 129993301 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13691771227136 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚274033554785280 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13691602227200 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 128199913 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 196340575 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚287725139009536 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚74841104 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚260681111630848 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚260707071852544 _step‚798 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130006510 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13693164378624 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 324572962 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13693164378624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13929917756416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274060806147072 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9699 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128212574 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _runtime‚ 7655.45983039 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚260707054813184 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚274060806147072 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128212574 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚274060824150016 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74848483 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 324552896 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274060824150016 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 324572962 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 % _timestamp‚1.7884030308223944e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13653683009852 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55158027 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55157509 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304306042843136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚304305814623232 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13930122036224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚287753988528640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13692995365376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚290375896866816 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 196360388 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128212572 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚520 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196360388 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228219904 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚274360754862396 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 196340324 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13653517888240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130005990 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 324552896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74848481 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13653517888240 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130005990 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 324552896 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚74848481 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚260707071852544 _step‚798 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130006510 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13693164378624 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 324572962 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13693164378624 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚13929917756416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚274060806147072 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 trainer/global_step‚9699 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128212574 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _runtime‚ 7655.45983039 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚260707054813184 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚274060806147072 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 128212574 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚274060824150016 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚74848483 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 324552896 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚274060824150016 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 324572962 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 % _timestamp‚1.7884030308223944e+09 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13653683009852 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55158027 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55157509 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚304306042843136 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚304305814623232 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚13930122036224 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚287753988528640 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13692995365376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚290375896866816 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 196360388 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128212572 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚520 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 196360388 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228219904 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚274360754862396 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 196340324žt{Àb2Î$- _wandbruntime‚7655  _runtime‚7655ddKÎ[Ï$Õ 1loss/SRqqt1/detection_loss‚1.7281880679668165 /loss/FBt2/detection_loss‚2.3801907924088566 /loss/FRt2/detection_loss‚2.9202491555895125 .loss/FRt3/detection_loss‚2.432652026414871 1loss/SRqqt4/detection_loss‚1.3361507699174702 trainer/global_step‚9699 .loss/FRt4/detection_loss‚1.933587333464876 _runtime‚7655.460925997 1loss/SRqqt1/assignment_loss‚1.168580614103406 1loss/FBt3/assignment_loss‚0.23578701437346833 0loss/SRqqt2/detection_loss‚4.208518916368485 % _timestamp‚1.7884030308241723e+09 _step‚799 /loss/FRt2/assignment_loss‚3.059896869318826 2loss/SRqqt3/assignment_loss‚1.4706430795057766 2loss/SRqqt4/assignment_loss‚0.9113052738654726 1loss/SRqqt3/detection_loss‚1.6917975553364124 /loss/FBt4/detection_loss‚0.9029764083051492 0loss/FRt3/assignment_loss‚2.7390711088569795 /loss/FBt3/detection_loss‚2.4811275137795343 1loss/SRqqt2/assignment_loss‚1.915085735420386 0loss/FBt1/assignment_loss‚0.4813886218209594 /loss/FRt1/detection_loss‚1.8068152350277493 /loss/FBt1/detection_loss‚2.7670878171920776 %loss/total_loss‚43.51007057780227 0loss/FRt4/assignment_loss‚1.4182918870702703 1loss/FBt4/assignment_loss‚0.18819815163722736 epoch‚8 /loss/FRt1/assignment_loss‚2.886764620667073 1loss/FBt2/assignment_loss‚0.44571600939060896ŠfËÒÐ$Ú /loss/FRt1/assignment_loss‚2.886764620667073 1loss/FBt2/assignment_loss‚0.44571600939060896 1loss/SRqqt1/detection_loss‚1.7281880679668165 /loss/FBt2/detection_loss‚2.3801907924088566 /ºAkËŽloss/FRt2/detection_loss‚2.9202491555895125 .loss/FRt3/detection_loss‚2.432652026414871 1loss/SRqqt4/detection_loss‚1.3361507699174702 trainer/global_step‚9699 .loss/FRt4/detection_loss‚1.933587333464876 _runtime‚7655.460925997 1loss/SRqqt1/assignment_loss‚1.168580614103406 1loss/FBt3/assignment_loss‚0.23578701437346833 0loss/SRqqt2/detection_loss‚4.208518916368485 % _timestamp‚1.7884030308241723e+09 _step‚799 /loss/FRt2/assignment_loss‚3.059896869318826 2loss/SRqqt3/assignment_loss‚1.4706430795057766 2loss/SRqqt4/assignment_loss‚0.9113052738654726 1loss/SRqqt3/detection_loss‚1.6917975553364124 /loss/FBt4/detection_loss‚0.9029764083051492 0loss/FRt3/assignment_loss‚2.7390711088569795 /loss/FBt3/detection_loss‚2.4811275137795343 1loss/SRqqt2/assignment_loss‚1.915085735420386 0loss/FBt1/assignment_loss‚0.4813886218209594 /loss/FRt1/detection_loss‚1.8068152350277493 /loss/FBt1/detection_loss‚2.7670878171920776 %loss/total_loss‚43.51007057780227 0loss/FRt4/assignment_loss‚1.4182918870702703 1loss/FBt4/assignment_loss‚0.18819815163722736 epoch‚8ŸE_ß‘Ñ$‚(:† àºãԧΉ® gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚145.595' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.119 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÿx˜Ò$‚(:  àºãÔúû”°$ disk./.usageGB‚233.5114860534668 disk.sda3.out‚ 8034.71484375 disk.nvme0n1.in‚ 400.03515625& disk.nvme0n1.out‚35463.29443359375 disk.sda2.in‚0 disk.drbd2266.out‚0$ memory_percent‚96.81229591369629" proc.memory.rssMB‚ 2694.4921875 disk.rbd1.in‚0 disk.sda.in‚ 0.06640625 disk.drbd1716.in‚0 disk.drbd1716.out‚0* disk./.usagePercent‚15.671151758235169 disk.drbd2080.in‚0" disk.nvme0c0n1.in‚ 400.03515625 proc.cpu.threads‚17 network.sent‚ 2.412945e+07 disk.drbd2266.in‚0 disk.drbd2080.out‚0( disk.nvme0c0n1.out‚35463.29443359375 disk.dm-1.out‚ 8034.71484375' proc.memory.availableMB‚ 522.2734375) proc.memory.percent‚16.44587516784668 cpu‚25.379295812958244 disk.rbd0.in‚ 0.0546875 disk.rbd1.out‚ 19.6474609375 disk.sda2.out‚0 disk.dm-1.in‚ 0.06640625 network.recv‚ 2.752236e+06 disk.rbd0.out‚ 0.32861328125 disk.dm-0.in‚ 400.03515625 disk.sda1.out‚0 disk.sda3.in‚ 0.06640625 disk.sda.out‚ 8034.71484375 disk.drbd2265.in‚0# disk.dm-0.out‚35463.29443359375 disk.sda1.in‚0 disk.drbd2265.out‚0Y¼j’Ó$‚(:‡ ïºãÔ—ã® gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚138.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.6658 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0æ‚‘,–Ô$‚(:‹  ïºãÔáŽì± network.recv‚ 2.759104e+06 disk.drbd2266.in‚0 disk.rbd0.out‚ 0.32861328125 disk.sda.out‚ 8036.48046875 disk.dm-1.in‚ 0.06640625 disk.drbd2080.out‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚35467.9873046875 disk.nvme0n1.in‚ 400.03515625 disk.drbd2265.out‚0 disk.dm-0.in‚ 400.03515625 disk.sda1.out‚0' proc.memory.availableMB‚ 521.6953125 disk.drbd2266.out‚0" disk.dm-0.out‚35467.9873046875 cpu‚25.378643187049587 disk.rbd0.in‚ 0.0546875 disk.sda3.out‚ 8036.48046875 disk.sda.in‚ 0.06640625 disk.sda2.in‚0) proc.memory.percent‚16.44587516784668* disk./.usagePercent‚15.671156878385853 disk.drbd2080.in‚0 disk.sda3.in‚ 0.06640625" disk.nvme0c0n1.in‚ 400.03515625 disk.sda2.out‚0 disk.drbd1716.in‚0" proc.memory.rssMB‚ 2694.4921875 network.sent‚ 2.4238588e+07 disk.rbd1.in‚0% disk.nvme0n1.out‚35467.9873046875 disk.drbd1716.out‚0 proc.cpu.threads‚17$ disk./.usageGB‚233.5115623474121 disk.dm-1.out‚ 8036.48046875 disk.drbd2265.in‚0 disk.rbd1.out‚ 19.7099609375$ memory_percent‚96.815824508666992Û_Ñ2Õ$- _wandbruntime‚7690  _runtime‚7690ï42ÈÖ$ _step‚800 ' lr-AdamW/pg1‚2.3633027119106882e-05 ' lr-AdamW/pg2‚2.3633027119106882e-05 trainer/global_step‚9749 $ _timestamp‚1.788403065715748e+09 _runtime‚7690.352595508l«PÛÍ×$Ç trainer/global_step‚9749 $ _timestamp‚1.788403065715748e+09 _runtime‚7690.352595508 _step‚800 ' lr-AdamW/pg1‚2.3633027119106882e-05 ' lr-AdamW/pg2‚2.3633027119106882e-05 °/¯2Ø$- _wandbruntime‚7690  _runtime‚7690lYó–í'Ù$çO XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13761273549824 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚289158465571840 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75209956 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 _step‚801 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚538 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13761442549760 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13761273549824 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 130634106 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚291792655269888 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 326146102 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚540 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 _runtime‚7690.353412659 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289158278568960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚275397005019136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚275699871289752 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚9749 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 197333277 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128832865 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128832865 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 197333277 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13761442549760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245010432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚261978270747648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 197313239 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55423612 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130633566 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚275397023022080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚289158278568960 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚275397005019136 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚275699871289752 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 trainer/global_step‚9749 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 197333277 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 128832865 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 128832865 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 197333277 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13761442549760 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245010432 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚261978270747648 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 197313239 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚55423612 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 130633566 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚275397023022080 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚291821544278016 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128845524 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130646353 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55429533 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚262004213929984 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13723154828260 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13722989706648 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚275424274383872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _step‚802 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚275424274383872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13762835701248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13762666688000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128845522 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚305822342679552 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 197333026 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚291821568218112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 326198614 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224025600 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14000798401536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 326178548 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 197353090 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75217331 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚9749 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 326178548 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 $ _timestamp‚1.788403066380919e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130646866 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55429022 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 128845524 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 130646353 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚55429533 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚262004213929984 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13723154828260 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13722989706648 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚275424274383872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 _step‚802 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚275424274383872 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13762835701248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13762666688000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 128845522 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚305822342679552 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 197333026 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚291821568218112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 326198614 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224025600 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14000798401536 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 326178548 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 197353090 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75217331 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 trainer/global_step‚9749 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 326178548 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 $ _timestamp‚1.788403066380919e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 130646866 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55429022 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚291821544278016 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199¢ˆÊA(2Þ$- _wandbruntime‚7691  _runtime‚7691Š_„]ß$× 2loss/SRqqt3/assignment_loss‚1.4501924187772803 2loss/SRqqt4/assignment_loss‚1.0107730723528274 0loss/SRqqt2/detection_loss‚2.346284907115133 /loss/FBt2/detection_loss‚3.8979862531026206 % _timestamp‚1.7884030663825634e+09 /loss/FRt1/detection_loss‚2.2958616328736317 /loss/FRt1/assignment_loss‚3.380816405018171 0loss/FBt1/assignment_loss‚0.2662259402076188 trainer/global_step‚9749 .loss/FBt1/detection_loss‚2.409649682896478 1loss/SRqqt3/detection_loss‚1.6888908393405102 _runtime‚7691.019343174 _step‚803 /loss/FRt4/detection_loss‚2.0152976903133095 1loss/SRqqt1/detection_loss‚1.9625667035579681 /loss/FBt3/detection_loss‚2.4864157140254974 /loss/FBt4/detection_loss‚0.6750076663692629 1loss/FBt4/assignment_loss‚0.16325724394430918 .loss/FRt2/detection_loss‚2.781397657468915 .loss/FRt3/detection_loss‚2.422906170531017 &loss/total_loss‚42.969642122917364 epoch‚8 /loss/FRt2/assignment_loss‚3.013008739799261 0loss/FRt4/assignment_loss‚1.5770960748195648 2loss/SRqqt1/assignment_loss‚1.6072808479746947 2loss/SRqqt2/assignment_loss‚1.7835787777837955 1loss/FBt2/assignment_loss‚0.11069899238523817 1loss/FBt3/assignment_loss‚0.39595845399890095 1loss/SRqqt4/detection_loss‚1.0708978617035632 0loss/FRt3/assignment_loss‚2.1575923765577922_:Dbà$Ü .loss/FBt1/detection_loss‚2.409649682896478 1loss/SRqqt3/detection_loss‚1.6888908393405102 _runtime‚7691.019343174 _step‚803 /loss/FRt4/detection_loss‚2.0152976903133095 1loss/SRqqt1/detection_loss‚1.9625667035579681 /loss/FBt3/detection_loss‚2.4864157140254974 /loss/FBt4/detection_loss‚0.6750076663692629 1loss/FBt4/assignment_loss‚0.16325724394430918 .loss/FRt2/detection_loss‚2.781397657468915 .loss/FRt3/detection_loss‚2.422906170531017 &loss/total_loss‚42.969642122917364 epoch‚8 /loss/FRt2/assignment_loss‚3.013008739799261 0loss/FRt4/assignment_loss‚1.5770960748195648 2loss/SRqqt1/assignment_loss‚1.6072808479746947 2loss/SRqqt2/assignment_loss‚1.7835787777837955 1loss/FBt2/assignment_loss‚0.11069899238523817 1loss/FBt3/assignment_loss‚0.39595845399890095 1loss/SRqqt4/detection_loss‚1.0708978617035632 0loss/FRt3/assignment_loss‚2.1575923765577922 2loss/SRqqt3/assignment_loss‚1.4501924187772803 2loss/SRqqt4/assignment_loss‚1.0107730723528274 0loss/SRqqt2/detection_loss‚2.346284907115133 /loss/FBt2/detection_loss‚3.8979862531026206 % _timestamp‚1.7884030663825634e+09 /loss/FRt1/detection_loss‚2.2958616328736317 /loss/FRt1/assignment_loss‚3.380816405018171 0loss/FBt1/assignment_loss‚0.2662259402076188 trainer/global_step‚9749£Úûÿ’á$‚(:‡ þºãÔÁ¶™® gpu.0.gpu‚29.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚124.046' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.8092 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ê5’·–â$‚(:‹  þºãÔÞÂÑ° disk.drbd2266.out‚0 disk.rbd0.in‚ 0.0546875" proc.memory.rssMB‚ 2694.4921875) proc.memory.percent‚16.44587516784668" disk.nvme0c0n1.in‚ 400.03515625 disk.dm-1.in‚ 0.06640625 disk.dm-0.in‚ 400.03515625 proc.cpu.threads‚17 disk.rbd1.in‚0 disk.drbd2265.out‚0 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.sda3.in‚ 0.06640625 disk.sda3.out‚ 8037.16015625( proc.memory.availableMB‚ 521.17578125 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 19.7099609375% disk.nvme0n1.out‚35468.2412109375$ disk./.usageGB‚233.5114860534668 disk.drbd1716.in‚0 disk.dm-1.out‚ 8037.16015625' disk.nvme0c0n1.out‚35468.2412109375 network.sent‚ 2.4243857e+07* disk./.usagePercent‚15.671151758235169 disk.drbd1716.out‚0 disk.rbd0.out‚ 0.3720703125" disk.dm-0.out‚35468.2412109375 disk.sda1.out‚0 disk.sda.out‚ 8037.16015625 network.recv‚ 2.762573e+06 disk.sda2.out‚0 disk.sda1.in‚0 disk.sda.in‚ 0.06640625$ memory_percent‚96.81899547576904 cpu‚25.378012921098787 disk.drbd2266.in‚0 disk.nvme0n1.in‚ 400.03515625ðÑéžã$‚(:’ »ãÔÞ·”® gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚138.866' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.773200000000003 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0CõBB’ä$‚(:‡  »ãÔÊú•² disk.drbd2080.out‚0& disk.nvme0n1.out‚35468.61376953125 disk.drbd2266.in‚0 disk.drbd2266.out‚0* disk./.usagePercent‚15.671151758235169 disk.sda1.out‚0 disk.nvme0n1.in‚ 400.03515625 disk.sda.out‚ 8037.4765625 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0$ memory_percent‚96.82097434997559 disk.rbd1.out‚ 19.8359375 disk.sda2.in‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2265.in‚0 disk.rbd0.in‚ 0.0546875 disk.dm-0.in‚ 400.03515625) proc.memory.percent‚16.44587516784668 disk.rbd0.out‚ 0.3720703125# disk.dm-0.out‚35468.61376953125' proc.memory.availableMB‚ 520.8515625 network.sent‚ 2.4352536e+07$ disk./.usageGB‚233.5114860534668 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 400.03515625 disk.drbd1716.in‚0" proc.memory.rssMB‚ 2694.4921875 network.recv‚ 2.768695e+06( disk.nvme0c0n1.out‚35468.61376953125 disk.sda.in‚ 0.06640625 cpu‚25.377366399731887 disk.sda3.out‚ 8037.4765625 disk.dm-1.out‚ 8037.4765625 disk.drbd2265.out‚0 proc.cpu.threads‚17 disk.sda1.in‚0 disk.sda3.in‚ 0.06640625”Jp‘å$‚(:† œ»ãÔÔÞŽ® gpu.0.gpu‚34.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚124.865' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.973 gpu.0.smClock‚1275 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ñ%€æ$‚(:õ œ»ãÔÙþò¯ disk.nvme0c0n1.in‚ 400.21875 disk.nvme0n1.in‚ 400.21875 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚ 0.43505859375# disk.nvme0n1.out‚35469.01953125) proc.memory.percent‚16.44587516784668 cpu‚25.376715986520576 network.recv‚ 2.772972e+06* disk./.usagePercent‚15.760475090999416 disk.sda.out‚ 8038.265625 disk.sda2.in‚0 disk.dm-1.in‚ 0.06640625 disk.dm-1.out‚ 8038.265625 disk.drbd2265.in‚0 disk.dm-0.in‚ 400.21875 disk.rbd1.out‚ 19.8359375% disk.nvme0c0n1.out‚35469.01953125 disk.sda2.out‚0$ memory_percent‚96.82114124298096 disk.dm-0.out‚35469.01953125 disk.sda1.out‚0 disk.sda.in‚ 0.06640625" proc.memory.rssMB‚ 2694.4921875 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.sda3.in‚ 0.06640625% disk./.usageGB‚234.84246826171875 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.sda3.out‚ 8038.265625 network.sent‚ 2.4358608e+07 disk.drbd1716.in‚0 disk.rbd0.in‚ 0.0546875( proc.memory.availableMB‚ 520.82421875 disk.sda1.in‚0 disk.rbd1.in‚0ª™=–2ç$- _wandbruntime‚7725  _runtime‚7725\ýŸ*Èè$ ' lr-AdamW/pg1‚2.2671000628101074e-05 ' lr-AdamW/pg2‚2.2671000628101074e-05 trainer/global_step‚9799 % _timestamp‚1.7884031012008598e+09 _runtime‚ 7725.83769213 _step‚804ß'_üÍé$Ç _step‚804 ' lr-AdamW/pg1‚2.2671000628101074e-05 ' lr-AdamW/pg2‚2.2671000628101074e-05 trainer/global_step‚9799 % _timestamp‚1.7884031012008598e+09 _runtime‚ 7725.83769213¤(mqè2ê$- _wandbruntime‚7725  _runtime‚7725ùdj²í'ë$çO JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 129465815 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚293238278740992 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14070323043328 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13791237481916 _step‚805 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 327771754 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 131275027 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 198325979 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75578804 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚276760473255936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55696745 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚277066502224960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚263275446903808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚277066684385724 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 198325979 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚276760473255936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75578806 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚290591605131264 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129465813 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13831113872384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚290591605131264 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚290591418128384 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13830944872448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚307308834211840 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 198305941 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13791072360512 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚290591605131264 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 129465813 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13831113872384 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.aúPŽÝ llocated‚290591605131264 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚290591418128384 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13830944872448 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚307308834211840 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 198305941 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13791072360512 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚276760473255936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55696745 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚277066502224960 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚263275446903808 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚277066684385724 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 198325979 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408¥£°»Ä2í$- _wandbruntime‚7726  _runtime‚7726;ý2àö&î$ðM JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚290620267647488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚ 7726.50802639 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 327824266 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 % _timestamp‚1.7884031018706682e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13792626646668 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131287770 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _step‚806 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131288265 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 198345792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚290620267647488 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 198345792 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚276787742620672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚277094016732812 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13832507023872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75586181 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129478474 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚263301373046784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 327824266 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 198325728 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55701589 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚307338936046080 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 129478472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚495 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚293267215629312 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 trainer/global_step‚9799 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚263301390086144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13832507023872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚277093834571840 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13792461525056 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚290620080631296 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 198325728 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚293267191689216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0ZM¯“û&ï$õM REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75586181 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 129478474 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚263301373046784 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 327824266 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 198325728 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚55701589 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚307338936046080 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 129478472 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚495 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚293267215629312 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 trainer/global_step‚9799 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚263301390086144 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13832507023872 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚277093834571840 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13792461525056 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚290620080631296 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 198325728 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚293267191689216 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚290620267647488 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚ 7726.50802639 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 327824266 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 % _timestamp‚1.7884031018706682e+09 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚13792626646668 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131287770 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _step‚806 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 131288265 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 198345792 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚290620267647488 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 198345792 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚276787742620672 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚277094016732812 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13832507023872 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416¦\˜H"2ð$- _wandbruntime‚7726  _runtime‚7726è…Ï7Zñ$Ô 1loss/FBt2/assignment_loss‚0.43014690820127655 /loss/FRt2/detection_loss‚2.9938483834266663 epoch‚8 _runtime‚7726.509068475 1loss/FBt4/assignment_loss‚0.16807693896048376 %loss/total_loss‚45.74125507498276 0loss/FRt3/assignment_loss‚3.1549284234642982 0loss/FRt4/assignment_loss‚1.7496150231787138 2loss/SRqqt1/assignment_loss‚1.6571099008010193 /loss/FRt3/detection_loss‚2.5719632491469384 /loss/FRt4/detection_loss‚1.5181559707437242 /loss/SRqqt3/detection_loss‚1.75863361927596 /loss/FBt2/detection_loss‚2.7182892948389052 /loss/FRt1/assignment_loss‚3.606627235607225 2loss/SRqqt4/assignment_loss‚0.9565919370032274 0loss/FBt3/assignment_loss‚0.6925114099867642 1loss/SRqqt1/detection_loss‚2.1007822941650045 /loss/FBt1/detection_loss‚2.3636953884905036 _step‚807 1loss/SRqqt2/assignment_loss‚2.031482446742685 .loss/FRt1/detection_loss‚2.065182259800867 1loss/SRqqt2/detection_loss‚2.4925513079291894 1loss/FBt1/assignment_loss‚0.46614463576497656 1loss/SRqqt4/detection_loss‚1.2836603357241705 .loss/FBt3/detection_loss‚2.786476125319799 /loss/FBt4/detection_loss‚0.6689021850410753 trainer/global_step‚9799 2loss/SRqqt3/assignment_loss‚1.6675381665202704 $ _timestamp‚1.788403101872307e+09 /loss/FRt2/assignment_loss‚3.838341634849022œí}€_ò$Ù 2loss/SRqqt1/assignment_loss‚1.6571099008010193 /loss/FRt3/detection_loss‚2.5719632491469384 /loss/FRt4/detection_loss‚1.5181559707437242 /loss/SRqqt3/detection_loss‚1.75863361927596 /loss/FBt2/detection_loss‚2.7182892948389052 /loss/FRt1/assignment_loss‚3.606627235607225 2loss/SRqqt4/assignment_loss‚0.9565919370032274 0loss/FBt3/assignment_loss‚0.6925114099867642 1loss/SRqqt1/detection_loss‚2.1007822941650045 /loss/FBt1/detection_loss‚2.3636953884905036 _step‚807 1loss/SRqqt2/assignment_loss‚2.031482446742685 .loss/FRt1/detection_loss‚2.065182259800867 1loss/SRqqt2/detection_loss‚2.4925513079291894 1loss/FBt1/assignment_loss‚0.46614463576497656 1loss/SRqqt4/detection_loss‚1.2836603357241705 .loss/FBt3/detection_loss‚2.786476125319799 /loss/FBt4/detection_loss‚0.6689021850410753 trainer/global_step‚9799 2loss/SRqqt3/assignment_loss‚1.6675381665202704 $ _timestamp‚1.788403101872307e+09 /loss/FRt2/assignment_loss‚3.838341634849022 1loss/FBt2/assignment_loss‚0.43014690820127655 /loss/FRt2/detection_loss‚2.9938483834266663 epoch‚8 _runtime‚7726.509068475 1loss/FBt4/assignment_loss‚0.16807693896048376 %loss/total_loss‚45.74125507498276 0loss/FRt3/assignment_loss‚3.1549284234642982 0loss/FRt4/assignment_loss‚1.7496150231787138§ñe3œó$‚(:‘ «»ãÔìɇ® gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚128.89' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.777999999999995 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­­$Ý:ô$‚(:ƒ  «»ãÔµ€è±# memory_percent‚96.8296766281128 proc.cpu.threads‚17 disk.rbd1.out‚ 19.8984375 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.rbd0.in‚ 0.0546875" proc.memory.rssMB‚ 2694.4921875 network.recv‚ 2.777723e+06* disk./.usagePercent‚15.760150985461067 disk.rbd1.in‚0 disk.nvme0n1.in‚ 403.96875 disk.dm-0.in‚ 403.96875) proc.memory.percent‚16.44587516784668( disk.nvme0c0n1.out‚36829.16552734375 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.nvme0c0n1.in‚ 403.96875 disk.sda.out‚ 8038.70703125 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚ 0.43505859375 disk.sda3.out‚ 8038.70703125( proc.memory.availableMB‚ 519.42578125 network.sent‚ 2.4364882e+07 disk.sda.in‚ 0.06640625# disk.dm-0.out‚36829.16552734375 disk.dm-1.out‚ 8038.70703125 disk.drbd2265.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.06640625% disk./.usageGB‚234.83763885498047& disk.nvme0n1.out‚36829.16552734375 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.dm-1.in‚K›QT 0.06640625 disk.sda1.in‚0 cpu‚25.376104031730815 disk.drbd2266.out‚0sÅ%1’õ$‚(:‡ º»ãÔ­Ü® gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚35.0 gpu.0.powerWatts‚148.842' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.7684 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0§·vv‚ö$‚(:÷ º»ãÔÛ€¥± disk.rbd1.in‚0& disk.nvme0n1.out‚36829.69287109375 disk.dm-1.out‚ 8039.3125" proc.memory.rssMB‚ 2694.4921875 network.recv‚ 2.783004e+06* disk./.usagePercent‚15.760146889340518 disk.rbd0.in‚ 0.0546875 disk.nvme0n1.in‚ 404.203125 network.sent‚ 2.447289e+07% disk./.usageGB‚234.83757781982422 disk.nvme0c0n1.in‚ 404.203125 disk.sda.out‚ 8039.3125 disk.sda2.out‚0 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚36829.69287109375 disk.drbd2266.in‚0 disk.drbd2265.in‚0$ memory_percent‚96.83024883270264) proc.memory.percent‚16.44587516784668 proc.cpu.threads‚17 disk.dm-0.in‚ 404.203125 disk.sda1.out‚0 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2080.out‚0# disk.dm-0.out‚36829.69287109375 disk.rbd1.out‚ 19.984375 disk.sda.in‚ 0.06640625 disk.sda2.in‚0 disk.drbd2265.out‚0 disk.rbd0.out‚ 0.498046875 disk.sda1.in‚0 disk.sda3.in‚ 0.06640625( proc.memory.availableMB‚ 519.33203125 disk.sda3.out‚ 8039.3125 disk.drbd1716.out‚0 disk.drbd2080.in‚0 cpu‚25.375478856122264(«—12÷$- _wandbruntime‚7761  _runtime‚7761uè+Çø$Á % _timestamp‚1.7884031367088356e+09 _runtime‚7761.345704002 _step‚808 & lr-AdamW/pg1‚2.172665097857098e-05 & lr-AdamW/pg2‚2.172665097857098e-05 trainer/global_step‚9849ÁÉ™'Ìù$Æ _runtime‚7761.345704002 _step‚808 & lr-AdamW/pg1‚2.172665097857098e-05 & lr-AdamW/pg2‚2.172665097857098e-05 trainer/global_step‚9849 % _timestamp‚1.7884031367088356e+09¨ª_ÛO2ú$- _wandbruntime‚7761  _runtime‚77612|¿ì'û$æO QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚278433133160168 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚292024744690688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚292024744690688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264572588981248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788403136709541e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 199298643 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292024557687808 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚278123959495680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130098765 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 329417446 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13900616195072 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130098765 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 131914593 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55966937 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13860709300324 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _runtime‚7761.346608578 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚278123941492736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚292024557687808 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 131914048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130098763 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 329397406 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 199318681 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13900785195008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚308825413484032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13900616195072 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚278123959495680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130098763 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 trainer/global_step‚9849 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75947656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75947654 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚294683950092288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13900785195008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚278433315320932 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13860544178920 _step‚809 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 199298643 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚308825174765056 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14141248612864 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚278433133160168 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚292024744690688 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚292024744690688 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚264572588981248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788403136709541e+09 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 199298643 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚292024557687808 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚278123959495680 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130098765 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 329417446 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚13900616195072 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130098765 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 131914593 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚55966937 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13860709300324 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _runtime‚7761.346608578 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚278123941492736 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚292024557687808 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 131914048 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130098763 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 329397406 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 199318681 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13900785195008 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚308825413484032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚13900616195072 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚278123959495680 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130098763 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 trainer/global_step‚9849 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚75947656 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚75947654 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚294683950092288 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13900785195008 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚278433315320932 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13860544178920 _step‚809 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 199298643 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚308825174765056 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14141248612864 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329449918 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚278460465507048 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚810 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚294712839100416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14142665679872 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚278460647668020 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 329429852 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _runtime‚ 7762.23351251 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75955033 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13902178346496 % _timestamp‚1.7884031375963557e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚264598532163584 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚278151210857472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚308855504780288 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130111424 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75955031 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131926854 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14142876251136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 199338494 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13902009333248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 199318430 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0¡8ÅÐHÿ$õM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.cuÕµ‘B³&rrent‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚75955031 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 131926854 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14142876251136 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 199338494 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13902009333248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 199318430 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 199338494 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13902009333248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130111424 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 329449918 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚278460465507048 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚810 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚294712839100416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14142665679872 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚278460647668020 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 329429852 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _runtime‚ 7762.23351251 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚75955033 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚13902178346496 % _timestamp‚1.7884031375963557e+09 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚264598532163584 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚278151210857472 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚526 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚308855504780288 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130111424 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0ªŽƒ1]2€%- _wandbruntime‚7762  _runtime‚7762oÃèZ%Ô 0loss/FRt3/assignment_loss‚3.1550317896736995 1loss/SRqqt3/assignment_loss‚1.209944676116316 /loss/FRt3/detection_loss‚2.4263288613822724 1loss/SRqqt4/detection_loss‚0.9844498624534983 /loss/FBt4/detection_loss‚0.7017790116133946 %loss/total_loss‚42.11560260253305 _step‚811 /loss/FRt2/detection_loss‚2.9464267261566652 1loss/SRqqt3/detection_loss‚1.6718255706407406 trainer/global_step‚9849 _runtime‚7762.234851134 0loss/FRt1/assignment_loss‚3.1418095588684083 1loss/SRqqt2/assignment_loss‚2.205038268776501 1loss/FBt4/assignment_loss‚0.15604031041473615 0loss/SRqqt2/detection_loss‚2.414620068143396 epoch‚8 1loss/SRqqt1/detection_loss‚1.7950136068794462 /loss/FBt2/detection_loss‚2.8235060274600983 2loss/SRqqt4/assignment_loss‚0.8066091562949097 0loss/FBt1/assignment_loss‚0.3117598217324182 0loss/FBt2/assignment_loss‚0.4726383085362613 0loss/FBt3/assignment_loss‚0.4049753784783974 /loss/FBt1/detection_loss‚2.3722854720221624 % _timestamp‚1.7884031375981245e+09 0loss/FRt2/assignment_loss‚3.8038558190868748 /loss/FRt4/assignment_loss‚1.117167924934963 2loss/SRqqt1/assignment_loss‚1.5751813772651884 .loss/FRt1/detection_loss‚2.057701687812805 /loss/FRt4/detection_loss‚1.2815926188849054 .loss/FBt3/detection_loss‚2.280020698904991Þц_‚%Ù _runtime‚7762.234851134 0loss/FRt1/assignment_loss‚3.1418095588684083 1loss/SRqqt2/assignment_loss‚2.205038268776501 1loss/FBt4/assignment_loss‚0.15604031041473615 0loss/SRqqt2/detection_loss‚2.414620068143396 epoch‚8 1loss/SRqqt1/detection_loss‚1.7950136068794462 /loss/FBt2/detection_loss‚2.8235060274600983 2loss/SRqqt4/assignment_loss‚0.8066091562949097 0loss/FBt1/assignment_loss‚0.3117598217324182 0loss/FBt2/assignment_loss‚0.4726383085362613 0loss/FBt3/assignment_loss‚0.4049753784783974 /loss/FBt1/detection_loss‚2.3722854720221624 % _timestamp‚1.7884031375981245e+09 0loss/FRt2/assignment_loss‚3.8038558190868748 /loss/FRt4/assignment_loss‚1.117167924934963 2loss/SRqqt1/assignment_loss‚1.5751813772651884 .loss/FRt1/detection_loss‚2.057701687812805 /loss/FRt4/detection_loss‚1.2815926188849054 .loss/FBt3/detection_loss‚2.280020698904991 0loss/FRt3/assignment_loss‚3.1550317896736995 1loss/SRqqt3/assignment_loss‚1.209944676116316 /loss/FRt3/detection_loss‚2.4263288613822724 1loss/SRqqt4/detection_loss‚0.9844498624534983 /loss/FBt4/detection_loss‚0.7017790116133946 %loss/total_loss‚42.11560260253305 _step‚811 /loss/FRt2/detection_loss‚2.9464267261566652 1loss/SRqqt3/detection_loss‚1.6718255706407406 trainer/global_step‚9849«C‰»¤ƒ%‚(:… É»ãÔŸª® gpu.0.gpu‚28.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚122.98' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.596 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0øvŒN†„%‚(:û É»ãÔ¢Ú˜± disk.nvme0n1.in‚ 404.203125 disk.dm-1.in‚ 0.06640625# proc.memory.availableMB‚518.875" proc.memory.rssMB‚ 2694.4921875% disk./.usageGB‚234.83717727661133 disk.drbd2266.in‚0 disk.drbd2266.out‚0) proc.memory.percent‚16.44587516784668 disk.rbd0.in‚ 0.0546875( disk.nvme0c0n1.out‚36830.20849609375 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.06640625 disk.rbd1.in‚0 disk.sda2.in‚0$ memory_percent‚96.83303833007812* disk./.usagePercent‚15.760120008549421 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 20.140625 disk.drbd1716.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚ 0.498046875 disk.sda3.out‚ 8039.7421875 disk.sda.out‚ 8039.7421875 disk.sda2.out‚0 network.recv‚ 2.787745e+06 disk.dm-0.in‚ 404.203125# disk.dm-0.out‚36830.20849609375 disk.dm-1.out‚ 8039.7421875 cpu‚25.37482101097178 network.sent‚ 2.4479086e+07 disk.drbd2265.out‚0 disk.nvme0c0n1.in‚ 404.203125& disk.nvme0n1.out‚36830.20849609375 disk.sda.in‚ 0.06640625 disk.drbd1716.in‚0 disk.sda1.in‚0#’…%‚(:‡ Ø»ãÔ˜»‘® gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚143.709' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.7418 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0te Œ†%‚(:  Ø»ãÔü®Ž±* disk./.usagePercent‚15.760128200790518 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.rbd1.out‚ 20.140625" proc.memory.rssMB‚ 2694.4921875) proc.memory.percent‚16.44587516784668 disk.sda2.out‚0 disk.dm-1.in‚ 0.06640625 disk.sda.in‚ 0.06640625 disk.drbd2080.out‚0 disk.rbd0.in‚ 0.0546875 disk.dm-0.in‚ 404.203125 disk.rbd1.in‚0 disk.drbd2265.out‚0 network.recv‚ 2.793356e+06 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.drbd2266.in‚0 disk.sda1.in‚0' disk.nvme0c0n1.out‚36830.5380859375$ memory_percent‚96.83382511138916 disk.rbd0.out‚ 0.498046875" disk.dm-0.out‚36830.5380859375 disk.nvme0c0n1.in‚ 404.203125 proc.cpu.threads‚17 disk.dm-1.out‚ 8040.54296875 disk.sda1.out‚0 disk.sda3.in‚ 0.06640625% disk.nvme0n1.out‚36830.5380859375 disk.sda3.out‚ 8040.54296875 disk.drbd2265.in‚0 network.sent‚ 2.4587496e+07% disk./.usageGB‚234.83729934692383 disk.drbd2266.out‚0( proc.memory.availableMB‚ 518.74609375 cpu‚25.374165798447088 disk.nvme0n1.in‚ 404.203125 disk.sda.out‚ 8040.54296875ÔþVÅ2‡%- _wandbruntime‚7796  _runtime‚7796£…Ž"ƈ%À _step‚812 & lr-AdamW/pg1‚2.080017497156815e-05 & lr-AdamW/pg2‚2.080017497156815e-05 trainer/global_step‚9899 $ _timestamp‚1.788403172348264e+09 _runtime‚7796.985107479ñBâˉ%Å $ _timestamp‚1.788403172348264e+09 _runtime‚7796.985107479 _step‚812 & lr-AdamW/pg1‚2.080017497156815e-05 & lr-AdamW/pg2‚2.080017497156815e-05 trainer/global_step‚9899¬V »2Š%- _wandbruntime‚7796  _runtime‚7796a‚·Âí'‹%çO XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56238676 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚13930181118732 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130731715 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 331043098 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 200291345 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 200311383 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130731713 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 132555182 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚558 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56238120 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13930015997328 _runtime‚7796.986018007 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚293457697247232 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚310341965854208 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14212151474688 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 200311383 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚279487427732480 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚296129597503488 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚293457884250112 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚293457697247232 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130731715 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 331023058 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚279487409729536 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚279799946256140 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132554624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚296129573563392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚310341725038080 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚279487427732480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚813 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130731713 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200291345 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚279487409729536 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚556 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14212368350720 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 trainer/global_step‚9899 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚13970456517632 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚293457884250112 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚265869748098048 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13970456517632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚13970456517632 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 130731715 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 331043098 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 200291345 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 200311383 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 130731713 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 132555182 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚558 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56238120 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13930015997328 _runtime‚7796.986018007 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚293457697247232 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚310341965854208 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14212151474688 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 20031138´û^3 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚279487427732480 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚296129597503488 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚293457884250112 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚293457697247232 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 130731715 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 331023058 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚279487409729536 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚279799946256140 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 132554624 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚296129573563392 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚310341725038080 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚279487427732480 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 _step‚813 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 130731713 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 200291345 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚279487409729536 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚556 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14212368350720 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199­}À³2%- _wandbruntime‚7797  _runtime‚7797M[€ö&Ž%ðM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56243964 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130744374 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130744372 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56243462 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 132567847 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 331055504 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200331196 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14213572265984 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331075570 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚504 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14213778642944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚ 7797.65200392 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚293486359750144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚279827096442256 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚265895691280384 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚293486546766336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7884031730146127e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76323881 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130744372 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 331075570 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130744374 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13971680655872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚502 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚279514697097216 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13971680655872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚310372289094656 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 _step‚814 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132567343 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13971849669120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚265895708319744 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚279514679094272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 200311132 h±œû&%õM YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚13931405161872 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 331075570 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚504 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14213778642944 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 _runtime‚ 7797.65200392 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚293486359750144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚279827096442256 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚265895691280384 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚293486546766336 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 % _timestamp‚1.7884031730146127e+09 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚76323881 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 130744372 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 331075570 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 130744374 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚13971680655872 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚502 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚279514697097216 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚13971680655872 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚310372289094656 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 _step‚814 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 132567343 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 PDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚13971849669120 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚265895708319744 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚279514679094272 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 200311132 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56243964 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 130744374 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 130744372 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚56243462 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 132567847 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 331055504 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 200331196 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14213572265984 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceSta½çç%tsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 332648710 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚311858517566464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚294890836806656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚281166577191348 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 201304085 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚280850877966336 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚267166907214848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133196441 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76685354 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚294891023809536 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14039958840320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13999487815736 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 201304085 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚267166924254208 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _step‚817 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚537 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 332648710 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚311858517566464 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚294890836806656 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚281166577191348 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 201304085 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚280850877966336 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚267166907214848 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 133196441 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚76685354 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚294891023809536 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14039958840320 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚13999487815736 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 201304085 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚267166924254208 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _step‚817 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚537 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133209170 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 332681156 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 201323898 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚297604157862912 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131377322 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 201303834 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚294919499309568 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76692733 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14284696489472 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 201323898 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚267192850397184 _step‚818 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14284488015360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚280878165334016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚281193727377464 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 trainer/global_step‚9949 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚294919686325760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚280878147331072 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14041520991744 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 332701222 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131377324 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚280878165334016 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332701222 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14000876980280 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7884032086093695e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚267192850397184 _step‚818 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14284488015360 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚280878165334016 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚281193727377464 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 trainer/global_step‚9949 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚294919686325760 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚280878147331072 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14041520991744 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 332701222 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 131377324 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚280878165334016 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 332701222 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14000876980280 I@DeviceStatsMoni å`}!tor.on_train_batch_end/allocation.large_pool.peak‚2460 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 % _timestamp‚1.7884032086093695e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133209170 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 332681156 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 201323898 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚297604157862912 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 131377322 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 201303834 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚294919499309568 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚76692733 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14284696489472 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 201323898 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 202296787 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚268464066331648 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14109630162944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56783970 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296323976366080 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚313374847873536 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚821 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14069124755548 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14068959634144 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚282533025965792 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚268464083371008il´N"®%ëO UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 131997615 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 131997613 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚56783428 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14109799162880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚282214364206080 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚313375092883968 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚296324163368960 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14109630162944 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14354200558080 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14109799162880 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 245010432 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚282214364206080 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 334294402 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 202276749 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77054206 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14353979487744 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚9999 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 202276749 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 202296787 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚268464066331648 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14109630162944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚56783970 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚296323976366080 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚313374847873536 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 _step‚821 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14069124755548 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14068959634144 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚282533025965792 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚268464083371008 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14070348798688 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚296352825885184 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14111023301120 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚313405416864768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚268490009513984 % _timestamp‚1.7884032439846523e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282560540473644 trainer/global_step‚9999 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 133850809 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132010274 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 334326874 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚296352825885184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77061581 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77061583 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56789226 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 202296536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132010274 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚296352638868992 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133850281 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132010272 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚299049805274112 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14070513920300 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚282241633570816 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334326874 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 334306808 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14070348798688 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚296352825885184 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14111023301120 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚313405416864768 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚268490009513984 % _timestamp‚1.7884032439846523e+09 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚282560540473644 trainer/global_step‚9999 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 133850809 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚526 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132010274 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 334326874 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚296352825885184 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77061581 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚77061583 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚56789226 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 202296536 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132010274 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚296352638868992 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 133850281 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132010272 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚299049805274112 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14070513920300 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚282241633570816 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 334326874 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 334306808 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132630563 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132630565 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚300466539737088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14179301485568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 134477955 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14138596573956 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132630563 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚283899839061764 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132630565 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚283577814439936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚297757302928384 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚283577814439936 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14424921910784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14179470485504 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚297757115925504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57054901 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚269761225448448 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚10049 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚297757302928384 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚314891437707776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚269761242487808 $ _timestamp‚1.788403279024961e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 335900014 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14179470485504 _runtime‚ 7903.66196546 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚300466515796992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 203269451 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚283899656901000 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217ŸZ´ñ'¾%ëO D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚297757115925504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57054901 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚269761225448448 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 trainer/global_step‚10049 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚297757302928384 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚314891437707776 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚269761242487808 $ _timestamp‚1.788403279024961e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 335900014 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14179470485504 _runtime‚ 7903.66196546 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚300466515796992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 203269451 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚283899656901000 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14179301485568 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 335920054 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 203269451 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _step‚825 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 132630563 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 132630565 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚300466539737088 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14179301485568 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 134477955 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14138596573956 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 132630563 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚283899839061764 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 132630565 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚283577814439936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚297757302928384 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚283577814439936 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14424921910784 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14179470485504 JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132643224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚509 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132643222 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14139985738708 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _runtime‚7904.330274385 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 132643222 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14180863636992 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14426547561472 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 134491143 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 203309302 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚283605101807616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚283605101807616 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚300495452685312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134490632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202182656 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 _step‚826 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 335952526 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚511 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57060201 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 335932460 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 335952526 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚297785965444608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚283927171408852 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14180694623744 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚314921774124032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚297785778428416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚300495428745216 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚283926989247880 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 132643224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57060710 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14426345378816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 203309302 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚314922000246784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 134491143 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 203309302 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚283605101807616 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚283605101807616 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚300495452685312 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 134490632 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202182656 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 _step‚826 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 335952526 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚511 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57060201 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 335932460 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 335952526 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚297785965444608 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚283927171408852 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14180694623744 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚314921774124032 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚297785778428416 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚300495428745216 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚283926989247880 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 OADeviceStatsMonitor.on_train_batch_end/active.largmCDþŸ e_pool.allocated‚ 132643224 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57060710 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14426345378816 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 203309302 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚314922000246784 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 132643224 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚509 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 132643222 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257ºôk—›2Â%- _wandbruntime‚7904  _runtime‚7904ëWYÃ%Ó 0loss/FBt2/assignment_loss‚0.2062276253430241 /loss/FRt2/detection_loss‚2.7976615501172617 /loss/FRt3/detection_loss‚2.1869163933472757 trainer/global_step‚10049 0loss/FRt4/assignment_loss‚1.8092012374024642 1loss/FBt1/assignment_loss‚0.46137659866774683 0loss/FRt1/assignment_loss‚3.4628624518712363 1loss/FBt3/assignment_loss‚0.08390389161638634 # _timestamp‚1.78840327969446e+09 2loss/SRqqt2/assignment_loss‚1.8417772510738084 .loss/FRt4/detection_loss‚1.822780292154404 0loss/SRqqt2/detection_loss‚2.434394446286288 1loss/SRqqt3/detection_loss‚1.6151620193998864 /loss/FBt4/detection_loss‚0.7724419760462993 %loss/total_loss‚43.88144006024258 1loss/SRqqt3/assignment_loss‚1.578112566427506 epoch‚8 _step‚827 1loss/SRqqt4/assignment_loss‚1.037299350534494 1loss/SRqqt1/detection_loss‚1.9285615345045013 1loss/SRqqt4/detection_loss‚1.1075806713448122 /loss/FBt1/detection_loss‚2.6282725291592732 1loss/FBt4/assignment_loss‚0.13421042732841482 .loss/FRt1/detection_loss‚2.013897681445408 /loss/FBt3/detection_loss‚2.8979462310671806 /loss/FRt2/assignment_loss‚3.659757375717163 /loss/FRt3/assignment_loss‚3.191959575200692 2loss/SRqqt1/assignment_loss‚1.3916208043276352 .loss/FBt2/detection_loss‚2.817515579859416 _runtime‚7904.331190092zÜý ^Ä%Ø .loss/FRt4/detection_loss‚1.822780292154404 0loss/SRqqt2/detection_loss‚2.434394446286288 1loss/SRqqt3/detection_loss‚1.6151620193998864 /loss/FBt4/detection_loss‚0.7724419760462993 %loss/total_loss‚43.88144006024258 1loss/SRqqt3/assignment_loss‚1.578112566427506 epoch‚8 _step‚827 1loss/SRqqt4/assignment_loss‚1.037299350534494 1loss/SRqqt1/detection_loss‚1.9285615345045013 1loss/SRqqt4/detection_loss‚1.1075806713448122 /loss/FBt1/detection_loss‚2.6282725291592732 1loss/FBt4/assignment_loss‚0.13421042732841482 .loss/FRt1/detection_loss‚2.013897681445408 /loss/FBt3/detection_loss‚2.8979462310671806 /loss/FRt2/assignment_loss‚3.659757375717163 /loss/FRt3/assignment_loss‚3.191959575200692 2loss/SRqqt1/assignment_loss‚1.3916208043276352 .loss/FBt2/detection_loss‚2.817515579859416 _runtime‚7904.331190092 0loss/FBt2/assignment_loss‚0.2062276253430241 /loss/FRt2/detection_loss‚2.7976615501172617 /loss/FRt3/detection_loss‚2.1869163933472757 trainer/global_step‚10049 0loss/FRt4/assignment_loss‚1.8092012374024642 1loss/FBt1/assignment_loss‚0.46137659866774683 0loss/FRt1/assignment_loss‚3.4628624518712363 1loss/FBt3/assignment_loss‚0.08390389161638634 # _timestamp‚1.78840327969446e+09 2loss/SRqqt2/assignment_loss‚1.8417772510738084»§ó´éÅ%‚(:’ мãÔò¡†® gpu.0.gpu‚36.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚148.312' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.662399999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0õÃf–Æ%‚(:‹  мãÔ´½š° disk.drbd2265.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 8044.66796875$ memory_percent‚96.83153629302979) proc.memory.percent‚16.44587516784668 proc.cpu.threads‚17 network.sent‚ 2.4941629e+07$ disk./.usageGB‚234.8374786376953 disk.drbd2080.out‚0" disk.nvme0c0n1.in‚ 404.20703125 disk.drbd2080.in‚0 disk.sda3.in‚ 0.06640625% disk.nvme0n1.out‚36836.0380859375 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.rbd1.out‚ 20.517578125 disk.nvme0n1.in‚ 404.20703125 disk.sda2.in‚0* disk./.usagePercent‚15.760140233144629 disk.dm-0.in‚ 404.20703125 disk.rbd1.in‚0 disk.sda.in‚ 0.06640625 disk.drbd1716.in‚0 disk.rbd0.out‚ 0.75634765625 disk.sda1.out‚0 disk.sda2.out‚0 disk.dm-1.out‚ 8044.66796875( proc.memory.availableMB‚ 519.12109375" proc.memory.rssMB‚ 2694.4921875" disk.dm-0.out‚36836.0380859375 network.recv‚ 2.832235e+06 disk.drbd2266.out‚0 disk.rbd0.in‚ 0.0546875' disk.nvme0c0n1.out‚36836.0380859375 disk.dm-1.in‚ 0.06640625 cpu‚25.369234818901667 disk.sda.out‚ 8044.66796875 disk.drbd2265.in‚0qíJp’Ç%‚(:‡ ß¼ãÔö®·® gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚119.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9466 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0H1O”È%‚(:‰  ß¼ãÔ¢èþ°" disk.nvme0c0n1.in‚ 404.20703125' disk.nvme0c0n1.out‚36836.5302734375 disk.drbd1716.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0$ memory_percent‚96.83535099029541 proc.cpu.threads‚17 network.recv‚ 2.838536e+06$ disk./.usageGB‚234.8375358581543 disk.drbd2266.in‚0 disk.sda.out‚ 8045.1484375 disk.dm-1.in‚ 0.06640625 disk.drbd2266.out‚0 disk.drbd2080.in‚0" disk.dm-0.out‚36836.5302734375 cpu‚25.368626849819144 disk.nvme0n1.in‚ 404.20703125% disk.nvme0n1.out‚36836.5302734375 disk.dm-1.out‚ 8045.1484375 disk.drbd2265.out‚0 disk.rbd0.in‚ 0.0546875 disk.rbd0.out‚ 0.75634765625 disk.sda.in‚ 0.06640625 disk.drbd1716.out‚0 disk.sda3.in‚ 0.06640625 disk.rbd1.out‚ 20.5498046875( proc.memory.availableMB‚ 518.49609375" proc.memory.rssMB‚ 2694.4921875 network.sent‚ 2.5050548e+07 disk.dm-0.in‚ 404.20703125) proc.memory.percent‚16.44587516784668 disk.sda2.in‚0 disk.sda3.out‚ 8045.1484375 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0* disk./.usagePercent‚15.760144073257642P™µTÉ%‚(:’ î¼ãÔì¸î­ gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚132.419' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.483800000000002 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 U»Ê%‚(:„  î¼ãÔáì¯ disk.sda2.out‚0 disk.dm-1.out‚ 8045.7578125 disk.sda1.in‚0( proc.memory.availableMB‚ 518.95703125& disk.nvme0c0n1.out‚36836.865234375 disk.drbd2266.in‚0 disk.sda1.out‚0) proc.memory.percent‚16.44587516784668 disk.sda3.out‚ 8045.7578125 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2265.in‚0 network.recv‚ 2.842067e+06$ disk./.usageGB‚234.8360137939453 disk.drbd2080.in‚0$ memory_percent‚96.83253765106201) disk./.usagePercent‚15.76004192625148 proc.cpu.threads‚17 disk.sda3.in‚ 0.06640625 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 404.20703125 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚ 0.8193359375" proc.memory.rssMB‚ 2694.4921875 disk.nvme0n1.in‚ 404.20703125$ disk.nvme0n1.out‚36836.865234375 cpu‚25.368034072507324 disk.sda.out‚ 8045.7578125 disk.drbd2080.out‚0 disk.rbd0.in‚ 0.0546875 disk.dm-0.in‚ 404.20703125! disk.dm-0.out‚36836.865234375 network.sent‚ 2.5055972e+07 disk.rbd1.out‚ 20.5498046875 disk.sda.in‚ 0.06640625É‚û-2Ë%- _wandbruntime‚7939  _runtime‚7939Mð¼€ÊÌ%Ä % _timestamp‚1.7884033143819005e+09 _runtime‚7939.018746286 _step‚828 ' lr-AdamW/pg1‚1.7276812180976958e-05 ' lr-AdamW/pg2‚1.7276812180976958e-05 trainer/global_step‚10099ø)ƒÏÍ%É % _timestamp‚1.7884033143819005e+09 _runtime‚7939.018746286 _step‚828 ' lr-AdamW/pg1‚1.7276812180976958e-05 ' lr-AdamW/pg2‚1.7276812180976958e-05 trainer/global_step‚10099¼UPò2Î%- _wandbruntime‚7939  _runtime‚7939L<Ä?ì'Ï%æO F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 337525666 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚284941300679680 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 trainer/global_step‚10099 _step‚829 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 204282191 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 % _timestamp‚1.7884033143826306e+09 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57327224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14248972808192 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14249141808128 _runtime‚ 7939.0196656 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 226136064 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133263515 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 204282191 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚299190255484928 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133263513 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133263515 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚301912163208192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14249141808128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚284941300679680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77791906 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚299190255484928 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 135119130 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57326685 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚316408071144448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271058384565248 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚316408297280512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚299190442487808 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚284941282676736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚299190442487808 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14207903270960 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚284941282676736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 337545706 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚285266287836208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14248972808192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚271058401604608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚301912187148288 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202195968 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 204282191 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚299190255484928 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133263513 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133263515 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚301912163208192 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14249141808128 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚284941300679680 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚77791906 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚299190255484928 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 135119130 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57326685 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚316408071144448 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚271058384565248 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚316408297280512 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚299190442487808 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚284941282676736 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚299190442487808 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14207903270960 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚284941282676736 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 337545706 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚285266287836208 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14248972808192 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚271058401604608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚301912187148288 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202195968 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 337578178 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77799281 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57332034 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135131315 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133276174 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚285293620183088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚299219105004032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚301941100096512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14209292435504 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚271084344786944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133276172 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚299219105004032 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚284968570044416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 135131828 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14209457557116 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133276174 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133276172 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚271084327747584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 % _timestamp‚1.7884033150531626e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 204302004 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚316438402866688 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚511 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14497535184384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚285293802344060 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚284968552041472 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 204281940 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 337578178 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14250534959616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 337558112 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 337578178 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚77799281 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57332034 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135131315 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133276174 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚285293620183088 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚299219105004032 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚301941100096512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14209292435504 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚271084344786944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 133276172 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚299219105004032 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚284968570044416 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 135131828 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14209457557116 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133276174 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133276172 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚271084327747584 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 % _timestamp‚1.7884033150531626e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 204302004 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚316438402866688 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚511 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14497535184384 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚285293802344060 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚284968552041472 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 204281940 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 337578178 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14250534959616 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 337558112 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300623395044352 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 _runtime‚7974.804695026 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 339171358 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78160754 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57598447 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚317924728363520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚286304768916480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14277375089368 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14567132523008 =6DeviceStý¨Q1 atsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚303357834559488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚286633100932180 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚272355543682048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚286304750913536 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133896465 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 205254855 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 % _timestamp‚1.7884033501677136e+09 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14566917744128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300623582047232 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 205274893 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 135758672 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133896463 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14318644130816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0w†kó'à%íO _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14318813130752 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 133896465 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚529 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚272355560721408 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 205274893 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚300623582047232 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚286304768916480 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚300623395044352 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 _runtime‚7974.804695026 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 339171358 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78160754 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57598447 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚317924728363520 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚286304768916480 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14277375089368 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14567132523008 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚303357834559488 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚286633100932180 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚272355543682048 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚286304750913536 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 133896465 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 205254855 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 % _timestamp‚1.7884033501677136e+09 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14566917744128 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚300623582047232 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 205274893 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 135758672 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 133896463 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14318644130816 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 133896463 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 339151318 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78160756 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚57597918 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚286632918771416 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14318644130816 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0Át¹Æ2á%- _wandbruntime‚7975  _runtime‚7975pn´÷&â%ñM \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚317955292551168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚286332020278272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚303386723567616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚7975.474786727 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14320037268992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚300652244563456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚286332020278272 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 $ _timestamp‚1.788403350837474e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135771351 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 339203830 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 339183764 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14320206282240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78168131 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57603220 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚286660251118296 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 trainer/global_step‚10149 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚303386747507712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14320037268992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133909124 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 205294706 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 205274642 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚286332038281216 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 339203830 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14320206282240 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚317955058039808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚272381486864384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133909122 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 135771849 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133909124 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _step‚834 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚300652057547264 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚498 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78168133 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14568545043456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚272381503903744 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14278764253912 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760~Ç:?©%ã%öM [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚286332038281216 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14278929375524 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚286332038281216 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 339203830 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14320206282240 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚317955058039808 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚272381486864384 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 133909122 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 135771849 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 133909124 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _step‚834 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚300652057547264 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚498 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78168133 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14568545043456 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚272381503903744 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14278764253912 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚317955292551168 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚286332020278272 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚303386723567616 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _runtime‚7975.474786727 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14320037268992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚300652244563456 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚286332020278272 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 $ _timestamp‚1.788403350837474e+09 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 135771351 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 339203830 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 339183764 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14320206282240 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78168131 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57603220 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚286660251118296 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 trainer/global_step‚10149 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚303386747507712 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14320037268992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 133909124 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 205294706 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 205274642 FDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚319441552497152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚304803481970688 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚302056534603776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 206267595 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚302056721606656 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206247557 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚287668219150336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚319441324263936 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚287668237153280 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚287668219150336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚304803458030592 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14637866233344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 206267595 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134529413 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136400278 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚287999731867388 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚273652702798848 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚302056721606656 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160ØF<¾ò'ò%ìO M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 134529413 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚302056534603776 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚525 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884033858437138e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚273652719838208 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14388315453440 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚57870672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _step‚837 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚78529604 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚78529606 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 206247557 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14388484453376 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14347012029180 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 206267595 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 134529413 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 136400278 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚287999731867388 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚273652702798848 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚302056721606656 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 134529415 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204293120 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 134529415 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚287999549706624 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚10199 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 340797010 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚319441552497152 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚304803481970688 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚302056534603776 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 206267595 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚302056721606656 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 206247557 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚287668219150336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚319441324263936 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚287668237153280 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚287668219150336 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚304803458030592 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14637866233344 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134542072 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14389708591616 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134542072 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚319471659117056 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚287695488515072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_ îµ3pool.freed‚ 206267344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 340829482 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚302085197106688 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57876309 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚304832394918912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 340829482 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 340809416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78536983 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57875786 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚288026882053504 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 206287408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚302085384122880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _step‚838 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134542074 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14348401193932 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 $ _timestamp‚1.788403386514645e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14389877604864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚302085384122880 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 340809416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14639288138240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78536981 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚287695506518016 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚287695488515072 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134542074 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14348236072320 trainer/global_step‚10199 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚287695506518016 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 206287408 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136412767 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 134542072 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14389708591616 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 134542072 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚319471659117056 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚287695488515072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 206267344 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 340829482 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚302085197106688 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚57876309 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚304832394918912 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 340829482 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 340809416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78536983 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚57875786 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚288026882053504 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 206287408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚302085384122880 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _step‚838 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 134542074 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14348401193932 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 $ _timestamp‚1.788403386514645e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14389877604864 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚302085384122880 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 340809416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14639288138240 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78536981 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚287695506518016 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚287695488515072 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 134542074 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14348236072320 trainer/global_step‚10199 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚287695506518016 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 206287408 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 136412767 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14709048370176 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚274949878955008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚289031705390080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚320958177752064 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135162365 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 207240259 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚289031687387136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚306249105441792 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135162363 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137040756 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 342402622 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚303489861166080 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135162363 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚303489674163200 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚289031687387136 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚303489861166080 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135162363 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚303489674163200 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚289031687387136 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14709048370176 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚274949878955008 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚289031705390080 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚320958177752064 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 135162365 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 207240259 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚289031687387136 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚306249105441792 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135162363 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137040756 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248Éêe’2ƒ&- _wandbruntime‚8047  _runtime‚8047ÊŸëø&„&òM SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 207260046 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _runtime‚8047.074922772 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚303518336666112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 342455134 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135175022 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14459379914240 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 342435068 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚303518523682304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚320988506271232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚320988269662720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 207260046 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚303518523682304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342455134 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58147698 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚289058956751872 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137052999 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78905831 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14459379914240 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14710463941120 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14417873012340 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78905833 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚274975805097984 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 207280110 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚289058974754816 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135175022 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 207280110 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14710251272704 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _step‚842 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944~'Û…ý&…&÷M VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58147698 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚289058956751872 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137052999 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚78905831 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14459379914240 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14710463941120 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14417873012340 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚78905833 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚274975805097984 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 207280110 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚289058974754816 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 135175022 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 207280110 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14710251272704 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 _step‚842 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 207260046 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _runtime‚8047.074922772 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚303518336666112 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 342455134 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135175022 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14459379914240 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 342435068 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚303518523682304 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚320988506271232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚320988269662720 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 207260046 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚303518523682304 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 342455134Êvü6M2†&- _wandbruntime‚8047  _runtime‚80474RÝZ‡&Ô /loss/FRt2/assignment_loss‚3.467288701307206 0loss/FBt3/assignment_loss‚0.5995794572601361 /loss/FRt1/detection_loss‚1.9479675902561775 /loss/FRt3/assignment_loss‚2.881773208578428 2loss/SRqqt1/assignment_loss‚1.4615852574352175 2loss/SRqqt3/assignment_loss‚1.5166133462236477 1loss/FBt4/assignment_loss‚0.17536657790248225 0loss/SRqqt2/detection_loss‚2.944085488716761 trainer/global_step‚10249 _step‚843 2loss/SRqqt2/assignment_loss‚2.1206002353380122 1loss/SRqqt4/detection_loss‚1.0806153992851895 epoch‚8 _runtime‚8047.076079955 0loss/FBt2/assignment_loss‚0.7683849506388469 0loss/SRqqt3/detection_loss‚1.643969225883484 /loss/FBt1/detection_loss‚2.3357516795396807 /loss/FBt2/detection_loss‚3.0177902976671853 /loss/FRt1/assignment_loss‚3.353687472776933 /loss/FRt2/detection_loss‚3.3275370058559237 1loss/SRqqt1/detection_loss‚1.8350589758095641 /loss/FBt4/detection_loss‚0.7783201042084807 /loss/FRt4/detection_loss‚1.5794223166324877 /loss/FBt3/detection_loss‚2.2013700859887257 /loss/FRt4/assignment_loss‚1.775073573806069 0loss/FBt1/assignment_loss‚0.5713143526925023 2loss/SRqqt4/assignment_loss‚1.1837335746855504 .loss/FRt3/detection_loss‚2.448775448318985 %loss/total_loss‚45.01566432680767 % _timestamp‚1.7884034224393344e+09@F‹Û{ˆ&Ù /loss/FRt2/assignment_loss‚3.467288701307206 0loss/FBt3/assignment_loss‚0.5995794572601361 /loss/FRt1/detection_loss‚1.9479675902561775 /loss/FRt3/assignment_loss‚2.881773208578428 2loss/SRqqt1/assignment_loss‚1.4615852574352175 2loss/SRqqt3/assignment_loss‚1.5166133462236477 1loss/FBt4/assignment_loss‚0.17536657790248225 0loss/SRqqt2/detection_loss‚2.944085488716761 trainer/global_step‚10249 _step‚843 2loss/SRqqt2/assignment_loss‚2.1206002353380122 1loss/SRqqt4/detection_loss‚1.0806153992851895 epoch‚8 _runtime‚8047.076079955 0loss/FBt2/assignment_loss‚0.7683849506388469 0loss/SRqqt3/detection_loss‚1.643969225883484 /loss/FBt1/detection_loss‚2.3357516795396807 /loss/FBt2/detection_loss‚3.0177902976671853 /loss/FRt1/assignment_loss‚3.353687472776933 /loss/FRt2/detection_loss‚3.327537005855…8øGä9237 1loss/SRqqt1/detection_loss‚1.8350589758095641 /loss/FBt4/detection_loss‚0.7783201042084807 /loss/FRt4/detection_loss‚1.5794223166324877 /loss/FBt3/detection_loss‚2.2013700859887257 /loss/FRt4/assignment_loss‚1.775073573806069 0loss/FBt1/assignment_loss‚0.5713143526925023 2loss/SRqqt4/assignment_loss‚1.1837335746855504 .loss/FRt3/detection_loss‚2.448775448318985 %loss/total_loss‚45.01566432680767 % _timestamp‚1.7884034224393344e+09ËNÒu±‰&‚(:’ æ½ãÔ”¿£® gpu.0.gpu‚24.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚111.353' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.270599999999998 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0zèr"‘Š&‚(:†  æ½ãÔñÁ ° network.recv‚ 2.881427e+06* disk./.usagePercent‚15.760352463390511 disk.sda3.in‚ 0.06640625& disk.nvme0n1.out‚36849.27880859375 disk.dm-1.out‚ 8052.015625' proc.memory.availableMB‚ 519.1171875) proc.memory.percent‚16.44587516784668 network.sent‚ 2.541126e+07% disk./.usageGB‚234.84064102172852 disk.rbd1.in‚0( disk.nvme0c0n1.out‚36849.27880859375 disk.sda2.in‚0 disk.rbd0.out‚ 0.88232421875 disk.sda1.in‚0 disk.sda.in‚ 0.06640625 disk.drbd2265.out‚0 disk.sda2.out‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2266.in‚0 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2694.4921875 proc.cpu.threads‚17 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 20.9267578125" disk.nvme0c0n1.in‚ 404.20703125 disk.drbd2080.in‚0# memory_percent‚96.8315601348877 disk.sda3.out‚ 8052.015625 disk.sda.out‚ 8052.015625 disk.drbd2265.in‚0 disk.rbd0.in‚ 0.0546875# disk.dm-0.out‚36849.27880859375 disk.sda1.out‚0 disk.nvme0n1.in‚ 404.20703125 disk.dm-0.in‚ 404.20703125 cpu‚25.36324618622767SJÜ’‹&‚(:‡ õ½ãÔù¶²® gpu.0.gpu‚35.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚117.447' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.4894 gpu.0.smClock‚1290 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0°öŽpŒ&‚(:‚  õ½ãÔòÕ° network.sent‚ 2.5519808e+07* disk./.usagePercent‚15.760352463390511 disk.sda.out‚ 8053.62109375 disk.rbd0.in‚ 0.0546875 cpu‚25.362662999231322 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.sda3.out‚ 8053.62109375" disk.nvme0c0n1.in‚ 404.20703125 disk.nvme0n1.in‚ 404.20703125 disk.drbd2080.in‚0 disk.drbd2265.out‚0 network.recv‚ 2.887653e+06 disk.sda1.in‚0# disk.nvme0n1.out‚36852.68359375 disk.dm-1.out‚ 8053.62109375 disk.dm-0.out‚36852.68359375% disk.nvme0c0n1.out‚36852.68359375 disk.drbd2266.in‚0 disk.drbd2265.in‚0 proc.cpu.threads‚17 disk.rbd1.out‚ 21.1689453125 disk.drbd1716.out‚0$ memory_percent‚96.83456420898438# proc.memory.availableMB‚518.625) proc.memory.percent‚16.44587516784668% disk./.usageGB‚234.84064102172852 disk.sda3.in‚ 0.06640625 disk.sda.in‚ 0.06640625 disk.sda2.in‚0 disk.drbd2080.out‚0" proc.memory.rssMB‚ 2694.4921875 disk.dm-0.in‚ 404.20703125 disk.rbd1.in‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2266.out‚0 disk.rbd0.out‚ 0.88232421875=œ¶P2&- _wandbruntime‚8081  _runtime‚8081•õß"ÊŽ&Ä trainer/global_step‚10299 % _timestamp‚1.7884034572785687e+09 _runtime‚8081.915427401 _step‚844 ' lr-AdamW/pg1‚1.4054186751196655e-05 ' lr-AdamW/pg2‚1.4054186751196655e-05²K–}Ï&É % _timestamp‚1.7884034572785687e+09 _runtime‚8081.915427401 _step‚844 ' lr-AdamW/pg1‚1.4054186751196655e-05 ' lr-AdamW/pg2‚1.4054186751196655e-05 trainer/global_step‚10299Ì´E¶2&- _wandbruntime‚8081  _runtime‚8081^î'‘&èO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚290395173626880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304922813722624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚322474529967104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 208232961 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚555 _runtime‚8081.916338931 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137682392 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚304923000725504 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚304923000725504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79267304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚276247021032448 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14485790544592 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 % _timestamp‚1.7884034572792563e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304922813722624 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 208232961 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218973184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚290395155623936 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚307694776793088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 208252999 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14527827098624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚290395173626880 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135795313 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135795315 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 344048314 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚290732993737804 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14779777114112 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14485955665996 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79267306 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚553 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14527827098624 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137681837 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚276247038071808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135795313 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 344028274 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14527658098688 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚304922813722624 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚322474529967104 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 208232961 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚555 _runtime‚8081.916338931 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 137682392 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚304923000725504 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚304923000725504 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79267304 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚276247021032448 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14485790544592 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 % _timestamp‚1.7884034572792563e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚304922813722624 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 208232961 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218973184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10299 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚290395155623936 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚307694776793088 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 208252999 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14527827098624 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚290395173626880 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 135795313 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 135795315 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 344048314 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚290732993737804 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14779777114112 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14485955665996 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79267306 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚553 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14527827098624 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 137681837 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚276247038071808 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 135795313 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 344028274 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14527658098688 JDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚8082.583627912 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14487344830748 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322504861894144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135807974 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135807972 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14781196092928 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14529220250112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚290422424988672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79274681 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚290760326084892 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 344080786 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚304951663241728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290760143923920 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚304951476225536 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚304951663241728 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14529220250112 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137694627 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 208252748 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚276272981254144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 208272812 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 344080786 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 344060720 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 208252748 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79274683 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 PBDevicSñ渔eStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 208272812 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58420483 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884034579463356e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14529051236864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚290422442991616 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚307723665801216 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14781406664192 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14487179709136S8ïý&•&÷M OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 135807974 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 135807972 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚537 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14781196092928 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14529220250112 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚290422424988672 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79274681 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚290760326084892 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 344080786 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚304951663241728 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚290760143923920 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚304951476225536 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚304951663241728 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14529220250112 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 137694627 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 208252748 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚276272981254144 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 208272812 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 344080786 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚539 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 344060720 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 208252748 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚79274683 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 208272812 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58420483 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 % _timestamp‚1.7884034579463356e+09 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14529051236864 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚290422442991616 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚307723665801216 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14781406664192 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14487179709136 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 _runtime‚8082.583627912 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14487344830748 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚322504861894144 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217ÎZF×2–&- _wandbruntime‚8082  _runtime‚8082$¾{C\—&Ö 1loss/SRqqt2/detection_loss‚2.7045809606025957 2loss/SRqqt2/assignment_loss‚1.6225845058416497 .loss/FRt2/detection_loss‚2.967091906070709 /loss/FRt3/assignment_loss‚2.420371647585522 1loss/SRqqt4/detection_loss‚0.9874428862431011 _step‚847 /loss/FRt1/assignment_loss‚3.298791392283006 0loss/FRt4/assignment_loss‚1.6362949109503202 2loss/SRqqt4/assignment_loss‚0.7555039962219288 /loss/FRt1/detection_loss‚1.8425728001807935 % _timestamp‚1.7884034579478905e+09 /loss/FRt2/assignment_loss‚3.904918118317922 2loss/SRqqt1/assignment_loss‚1.0765570285078139 0loss/FBt3/assignment_loss‚0.5080005457624793 0loss/FBt4/assignment_loss‚0.1825199732168986 .loss/FRt3/detection_loss‚2.233544130555608 /loss/FBt1/detection_loss‚2.1584986746311188 /loss/FBt3/detection_loss‚2.9442914485931397 /loss/FBt4/detection_loss‚0.5930764795076556 /loss/FRt4/detection_loss‚1.6650005916266568 1loss/SRqqt1/detection_loss‚1.7144969642162322 1loss/SRqqt3/detection_loss‚1.6568751321597532 %loss/total_loss‚42.36116653377091 epoch‚8 _runtime‚8082.584625263 2loss/SRqqt3/assignment_loss‚1.2419137332588435 1loss/FBt1/assignment_loss‚0.09289703170070425 trainer/global_step‚10299 0loss/FBt2/assignment_loss‚0.3402056941767076 /loss/FBt2/detection_loss‚3.8131359815597534)™Pýa˜&Û 2loss/SRqqt2/assignment_loss‚1.6225845058416497 .loss/FRt2/detection_loss‚2.967091906070709 /loss/FRt3/assignment_loss‚2.420371647585522 1loss/SRqqt4/detection_loss‚0.9874428862431011 _step‚847 /loss/FRt1/assignment_loss‚3.298791392283006 0loss/FRt4/assignment_loss‚1.6362949109503202 2loss/SRqqt4/assignment_loss‚0.7555039962219288 /loss/FRt1/detection_loss‚1.8425728001807935 % _timestamp‚1.7884034579478905e+09 /loss/FRt2/assignment_loss‚3.904918118317922 2loss/SRqqt1/assignment_loss‚1.0765570285078139 0loss/FBt3/assignment_loss‚0.5080005457624793 0loss/FBt4/assignment_loss‚0.1825199732168986 .loss/FRt3/detection_loss‚2.233544130555608 /loss/FBt1/detection_loss‚2.1584986746311188 /loss/FBt3/detection_loss‚2.9442914485931397 /loss/FBt4/detection_loss‚0.5930764795076556 /loss/FRt4/detection_loss‚1.6650005916266568 1loss/SRqqt1/detection_loss‚1.7144969642162322 1loss/SRqqt3/detection_loss‚1.6568751321597532 %loss/total_loss‚42.36116653377091 epoch‚8 _runtime‚8082.584625263 2loss/SRqqt3/assignment_loss‚1.2419137332588435 1loss/FBt1/assignment_loss‚0.09289703170070425 trainer/global_step‚10299 0loss/FBt2/assignment_loss‚0.3402056941767076 /loss/FBt2/detection_loss‚3.8131359815597534 1loss/SRqqt2/detection_loss‚2.7045809606025957Ï­‰ž-’™&‚(:‡ „¾ãÔ¢‘ú­ gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚144.287' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.8574 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0$ì\Ž—š&‚(:Œ  „¾ãÔ„Åœ° disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.rbd0.in‚ 0.0546875 cpu‚25.362087641284504* disk./.usagePercent‚15.760351183352842$ disk./.usageGB‚234.8406219482422 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda1.out‚0( proc.memory.availableMB‚ 518.89453125 disk.sda3.out‚ 8054.3046875 disk.rbd1.out‚ 21.1689453125" disk.nvme0c0n1.in‚ 404.20703125( disk.nvme0c0n1.out‚36852.99853515625) proc.memory.percent‚16.44587516784668 network.sent‚ 2.5525882e+07 disk.drbd2080.out‚0 disk.rbd0.out‚ 0.88232421875# disk.dm-0.out‚36852.99853515625 disk.rbd1.in‚0 disk.nvme0n1.in‚ 404.20703125 proc.cpu.threads‚17 network.recv‚ 2.891903e+06 disk.drbd2265.in‚0& disk.nvme0n1.out‚36852.99853515625 disk.dm-0.in‚ 404.20703125 disk.sda.in‚ 0.06640625" proc.memory.rssMB‚ 2694.4921875 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 8054.3046875 disk.sda1.in‚0 disk.dm-1.in‚ 0.06640625 disk.sda3.in‚ 0.06640625 disk.sda.out‚ 8054.3046875$ memory_percent‚96.83291912078857¯‚‘›&‚(:† “¾ãÔúÓ•® gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚145.595' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.119 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ߤ‰»“œ&‚(:ˆ  “¾ãÔú¤ú° disk.sda3.out‚ 8054.8203125% disk.nvme0n1.out‚36853.5771484375 disk.dm-1.in‚ 0.06640625 network.recv‚ 2.898504e+06 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚ 0.0546875 disk.dm-0.in‚ 404.20703125" disk.dm-0.out‚36853.5771484375 disk.sda1.in‚0 disk.rbd1.out‚ 21.201171875 network.sent‚ 2.5634668e+07' disk.nvme0c0n1.out‚36853.5771484375( proc.memory.availableMB‚ 519.23828125* disk./.usagePercent‚15.760359631601473% disk./.usageGB‚234.84074783325195 disk.drbd2266.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 cpu‚25.36150670020444 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.06640625 disk.sda.out‚ 8054.8203125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.in‚ 0.06640625 disk.nvme0n1.in‚ 404.20703125 disk.drbd1716.out‚0) proc.memory.percent‚16.44587516784668 disk.rbd0.out‚ 0.88232421875" disk.nvme0c0n1.in‚ 404.20703125 disk.dm-1.out‚ 8054.8203125" proc.memory.rssMB‚ 2694.4921875$ memory_percent‚96.83082103729248Q#‘&‚(:† ¢¾ãÔ÷·û­ gpu.0.gpu‚0.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚134.022' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.8044 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0É®pt‚ž&‚(:÷ ¢¾ãÔÝŸˆ° disk.dm-0.in‚ 404.23046875" disk.dm-0.out‚36871.5869140625 network.sent‚ 2.5640144e+07% disk./.usageGB‚234.84780883789062 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.sda1.out‚0) proc.memory.percent‚16.44587516784668 disk.rbd0.in‚ 0.0546875 disk.sda3.in‚ 0.06640625 disk.sda3.out‚ 8057.84375 proc.cpu.threads‚17" disk.nvme0c0n1.in‚ 404.23046875' disk.nvme0c0n1.out‚36871.5869140625 network.recv‚ 2.902263e+06) disk./.usagePercent‚15.76083350154735 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.rbd0.out‚ 0.88232421875" memory_percent‚96.832275390625 cpu‚25.360938486622306 disk.sda.in‚ 0.06640625 proc.memory.availableMB‚519" proc.memory.rssMB‚ 2694.4921875 disk.rbd1.in‚0 disk.rbd1.out‚ 21.201171875 disk.nvme0n1.in‚ 404.23046875% disk.nvme0n1.out‚36871.5869140625 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.dm-1.out‚ 8057.84375 disk.sda.out‚ 8057.84375 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0‘ˆ2Ÿ&- _wandbruntime‚8117  _runtime‚8117UªÚÈ & ' lr-AdamW/pg1‚1.3296802063710506e-05 ' lr-AdamW/pg2‚1.3296802063710506e-05 trainer/global_step‚10349 # _timestamp‚1.78840349287045e+09 _runtime‚8117.507334022 _step‚848Ħ 0Í¡&Ç ' lr-AdamW/pg1‚1.3296802063710506e-05 ' lr-AdamW/pg2‚1.3296802063710506e-05 trainer/global_step‚10349 # _timestamp‚1.78840349287045e+09 _runtime‚8117.507334022 _step‚848ÐÕ‰u2¢&- _wandbruntime‚8117  _runtime‚8117~gì'£&æO QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 209245701 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79636156 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14850708614144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136428265 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚277544180149248 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14597329421312 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚292099442512248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79636154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58687566 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚306356140284928 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 209225663 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14850917101568 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚292099624673012 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚323991341305856 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚309140400264192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 345673966 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚291758641863680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 138323722 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚291758623860736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚306355953282048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚10349 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14597498421248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14555262363000 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136428263 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚291758623860736 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚306355953282048 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14597329421312 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 _step‚849XüÁÄ ¤&ëO O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14597329421312 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 _step‚849 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 209245701 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚79636156 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14850708614144 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 136428265 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚277544180149248 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14597329421312 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚292099442512248 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚79636154 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58687566 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚30635614028492ÿ 7Š-8 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 209225663 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚14850917101568 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚292099624673012 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚552 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚323991341305856 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚309140400264192 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 345673966 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚291758641863680 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 138323722 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚291758623860736 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚306355953282048 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 trainer/global_step‚10349 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14597498421248 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14555262363000 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 136428263 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚291758623860736 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚306355953282048Ñž%?ç2¥&- _wandbruntime‚8118  _runtime‚8118ˆKBø&¦&òM EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884034935334363e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136440922 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 345706438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14598722559488 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚306384615784960 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚324021672757760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 345686372 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚291785893225472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚291785911228416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚324021444537856 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138335819 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14556651527544 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚309169313212416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚291785911228416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228219904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79643531 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚8118.170845717 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚306384802801152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 345686372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136440924 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14598891572736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 209245450 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 209265514 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚277570123331584 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚306384802801152 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14556816649156 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 345706438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚291785893225472 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14852131325440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0™µ¨†ý&§&÷M _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚309169313212416 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 204279808 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚291785911228416 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 228219904 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚79643531 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚8118.170845717 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚306384802801152 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 345686372 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 136440924 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14598891572736 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 209245450 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 209265514 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚277570123331584 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚306384802801152 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚529 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14556816649156 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 345706438 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚291785893225472 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14852131325440 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884034935334363e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 136440922 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 345706438 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14598722559488 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚306384615784960 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚324021672757760 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 345686372 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚291785893225472 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚291785911228416 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚324021444537856 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138335819 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14556651527544 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000ÒÑs™2¨&- _wandbruntime‚8118  _runtime‚8118âŽ1X©&Ò /loss/FRt3/assignment_loss‚3.035091906785965 1loss/SRqqt2/assignment_loss‚1.470841859281063 /loss/FRt1/detection_loss‚2.2255496778050246 1loss/SRqqt3/detection_loss‚1.9176485521925821 .loss/FBt2/detection_loss‚2.765773743391037 2loss/SRqqt3/assignment_loss‚1.5465887762192223 /loss/FRt2/detection_loss‚3.4363362278257097 /loss/FRt4/detection_loss‚1.7194677481150373 &loss/total_loss‚48.020795809040315 trainer/global_step‚10349 2loss/FBt1/assignment_loss‚0.058346979320049286 0loss/SRqqt2/detection_loss‚2.894413638114929 /loss/FRt4/assignment_loss‚1.551036612784609 0loss/FBt4/assignment_loss‚0.1916796630160725 /loss/FRt3/detection_loss‚2.2366524585767795 0loss/FRt2/assignment_loss‚3.5751362414587113 /loss/FBt3/detection_loss‚2.0484456649193397 _runtime‚8118.172097048 _step‚851 0loss/FRt1/assignment_loss‚2.9447539193289622 2loss/SRqqt1/assignment_loss‚1.9593249357960842 /loss/FBt2/assignment_loss‚0.658937452524242 /loss/FBt4/detection_loss‚0.6219162432834098 epoch‚8 2loss/SRqqt4/assignment_loss‚0.9320876264843989 0loss/SRqqt1/detection_loss‚1.538816748117959 /loss/FBt3/assignment_loss‚0.273397101304139 /loss/SRqqt4/detection_loss‚1.08676605550824 .loss/FBt1/detection_loss‚7.331785976886749 % _timestamp‚1.7884034935353408e+09B{È]ª&× 2loss/SRqqt4/assignment_loss‚0.9320876264843989 0loss/SRqqt1/detection_loss‚1.538816748117959 /loss/FBt3/assignment_loss‚0.273397101304139 /loss/SRqqt4/detection_loss‚1.08676605550824 .loss/FBt1/detection_loss‚7.331785976886749 % _timestamp‚1.7884034935353408e+09 /loss/FRt3/assignment_loss‚3.035091906785965 1loss/SRqqt2/assignment_loss‚1.470841859281063 /loss/FRt1/detection_loss‚2.2255496778050246 1loss/SRqqt3/detection_loss‚1.9176485521925821 .loss/FBt2/detection_loss‚2.765773743391037 2loss/SRqqt3/assignment_loss‚1.5465887762192223 /loss/FRt2/detection_loss‚3.4363362278257097 /loss/FRt4/detection_loss‚1.7194677481150373 &loss/total_loss‚48.020795809040315 trainer/global_step‚10349 2loss/FBt1/assignment_loss‚0.058346979320049286 0loss/SRqqt2/detection_loss‚2.894413638114929 /loss/FRt4/assignment_loss‚1.551036612784609 0loss/FBt4/assignment_loss‚0.1916796630160725 /loss/FRt3/detection_loss‚2.2366524585767795 0loss/FRt2/assignment_loss‚3.5751362414587113 /loss/FBt3/detection_loss‚2.0484456649193397 _runtime‚8118.172097048 _step‚851 0loss/FRt1/assignment_loss‚2.9447539193289622 2loss/SRqqt1/assignment_loss‚1.9593249357960842 /loss/FBt2/assignment_loss‚0.658937452524242 /loss/FBt4/detection_loss‚0.6219162432834098 epoch‚8ÓñÜãæ«&‚(:’ ±¾ãÔÔû™® gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚35.0 gpu.0.powerWatts‚148.016' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.603199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ö1°)†¬&‚(:û ±¾ãÔç¬Ô° network.sent‚ 2.5646239e+07 disk.drbd2265.in‚0 disk.dm-0.out‚ 36878.34375 disk.rbd1.out‚ 21.443359375 disk.nvme0n1.in‚ 404.23046875 cpu‚25.36033201284745 disk.sda.in‚ 0.06640625" proc.memory.rssMB‚ 2694.4921875% disk./.usageGB‚234.84472274780273 disk.sda.out‚ 8059.64453125 disk.dm-1.out‚ 8059.64453125 disk.drbd2266.in‚0 disk.rbd0.in‚ 0.0546875 disk.nvme0n1.out‚ 36878.34375) proc.memory.percent‚16.44587516784668 disk.sda2.in‚0 disk.sda3.in‚ 0.06640625 network.recv‚ 2.906687e+06 disk.drbd1716.in‚0 disk.drbd2080.out‚0$ memory_percent‚96.83241844177246 disk.sda2.out‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚ 0.88232421875 disk.sda1.in‚0 disk.rbd1.in‚0" disk.nvme0c0n1.out‚ 36878.34375* disk./.usagePercent‚15.760626391452151 disk.dm-0.in‚ 404.23046875" disk.nvme0c0n1.in‚ 404.23046875' proc.memory.availableMB‚ 518.9765625 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 8059.64453125tÊÚ­&‚(:’ À¾ãÔË£›® gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚117.311' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.462200000000003 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0”"8€¸®&‚(:„  À¾ãÔÎÔ¸± disk.nvme0n1.in‚ 404.23046875 disk.sda2.out‚0 disk.drbd2080.out‚0) proc.memory.percent‚16.44587516784668 proc.cpu.threads‚17* disk./.'u—×usagePercent‚15.760622551339138 disk.sda3.out‚ 8060.3671875 disk.dm-1.in‚ 0.06640625 disk.drbd2265.in‚0" disk.nvme0c0n1.in‚ 404.23046875% disk.nvme0n1.out‚36878.6845703125 disk.drbd1716.in‚0 disk.drbd2265.out‚0 network.sent‚ 2.5754555e+07' disk.nvme0c0n1.out‚36878.6845703125 disk.sda.in‚ 0.06640625 disk.sda.out‚ 8060.3671875 disk.sda2.in‚0" disk.dm-0.out‚36878.6845703125$ memory_percent‚96.83218002319336& proc.memory.availableMB‚ 519.015625 cpu‚25.3597266430868% disk./.usageGB‚234.84466552734375 disk.sda3.in‚ 0.06640625 disk.drbd1716.out‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd0.out‚ 0.88232421875 network.recv‚ 2.91262e+06 disk.rbd1.in‚0 disk.dm-1.out‚ 8060.3671875 disk.drbd2266.in‚0" proc.memory.rssMB‚ 2694.4921875 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚ 0.0546875 disk.dm-0.in‚ 404.23046875 disk.rbd1.out‚ 21.443359375x˜LB2¯&- _wandbruntime‚8152  _runtime‚8152 f¤ÀÊ°&Ä trainer/global_step‚10399 % _timestamp‚1.7884035280065398e+09 _runtime‚8152.643391691 _step‚852 ' lr-AdamW/pg1‚1.2559047786630467e-05 ' lr-AdamW/pg2‚1.2559047786630467e-05N¬†zϱ&É ' lr-AdamW/pg1‚1.2559047786630467e-05 ' lr-AdamW/pg2‚1.2559047786630467e-05 trainer/global_step‚10399 % _timestamp‚1.7884035280065398e+09 _runtime‚8152.643391691 _step‚852Ô‡°¿¤2²&- _wandbruntime‚8152  _runtime‚8152ß^°Éî'³&èO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚325507713487360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137061213 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚278841356305408 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _step‚853 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚293122110100480 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137061213 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58958907 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚325507948012032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚8152.644391267 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚293122110100480 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚293122092097536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 347299618 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 138963913 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14921665811968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚278841339266048 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚293122092097536 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14667000743936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚293466255608220 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138963377 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚310586071615488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58958373 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚307789092841472 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14667000743936 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160哵“ó'´&íO ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚325507948012032 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 _runtime‚8152.644391267 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚293122110100480 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚293122092097536 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 347299618 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 138963913 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14921665811968 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚278841339266048 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚293122092097536 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14667000743936 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚293466255608220 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 138963377 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚310586071615488 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚58958373 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚307789092841472 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14667000743936 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚325507713487360 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 137061213 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚278841356305408 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _step‚853 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚293122110100480 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137061213 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚58958907Õ¬zµ¬2µ&- _wandbruntime‚8153  _runtime‚8153ö»™ø&¶&òM OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 210258216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137073874 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚293493405794336 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58964378 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚310614960623616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚293493587955308 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14668562895360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14923083957760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 210238152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚278867282448384 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347332090 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14668393882112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80012383 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _step‚854 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14626123345952 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138976237 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚293149361462272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137073872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14668562895360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347332090 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 trainer/global_step‚10399 % _timestamp‚1.7884035288920734e+09 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14923294529024 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257¼]þUÛ·&÷M TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14668393882112 B8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.smalT{æ"$l_pool.allocated‚ 417333248 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚293149361462272 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137073872 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14668562895360 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 347332090 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 trainer/global_step‚10399 % _timestamp‚1.7884035288920734e+09 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚14923294529024 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 210258216 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137073874 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚293493405794336 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚58964378 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚310614960623616 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚293493587955308 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14668562895360 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14923083957760 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 210238152 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚278867282448384 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 347332090 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14668393882112 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80012383 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _step‚854 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14626123345952 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 138976237 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 PZ¹&Ô &loss/total_loss‚43.522665261252996 1loss/SRqqt4/assignment_loss‚1.354161686905836 1loss/FBt3/assignment_loss‚0.24793876514861304 /loss/FRt2/detection_loss‚2.8087154369096496 _runtime‚8153.530546567 0loss/SRqqt3/detection_loss‚1.929755019822291 0loss/FRt4/assignment_loss‚1.5140435898303986 /loss/FRt1/detection_loss‚2.0013557220735247 /loss/FRt3/detection_loss‚2.5846097618341446 .loss/FBt1/detection_loss‚2.076553149656816 epoch‚8 _step‚855 0loss/FRt1/assignment_loss‚2.9856931340126764 /loss/FBt2/assignment_loss‚0.619764906132909 1loss/SRqqt4/detection_loss‚1.0855848885871269 /loss/FBt2/detection_loss‚2.4100465224339414 0loss/FRt2/assignment_loss‚3.6083801404849902 /loss/FRt3/assignment_loss‚2.231826163828373 2loss/SRqqt1/assignment_loss‚1.4950327731950104 2loss/SRqqt2/assignment_loss‚1.9286396061932598 0loss/FBt1/assignment_loss‚0.3641790388447403 0loss/FBt4/assignment_loss‚0.2645624897324126 1loss/SRqqt1/detection_loss‚1.7556631491993957 1loss/SRqqt2/detection_loss‚2.6151797429279044 trainer/global_step‚10399 % _timestamp‚1.7884035288938053e+09 2loss/SRqqt3/assignment_loss‚1.4303420848612274 .loss/FRt4/detection_loss‚1.643550098836422 -loss/FBt3/detection_loss‚3.91525074839592 /loss/FBt4/detection_loss‚0.6518366414054138²sq_º&Ù 0loss/FRt1/assignment_loss‚2.9856931340126764 /loss/FBt2/assignment_loss‚0.619764906132909 1loss/SRqqt4/detection_loss‚1.0855848885871269 /loss/FBt2/detection_loss‚2.4100465224339414 0loss/FRt2/assignment_loss‚3.6083801404849902 /loss/FRt3/assignment_loss‚2.231826163828373 2loss/SRqqt1/assignment_loss‚1.4950327731950104 2loss/SRqqt2/assignment_loss‚1.9286396061932598 0loss/FBt1/assignment_loss‚0.3641790388447403 0loss/FBt4/assignment_loss‚0.2645624897324126 1loss/SRqqt1/detection_loss‚1.7556631491993957 1loss/SRqqt2/detection_loss‚2.6151797429279044 trainer/global_step‚10399 % _timestamp‚1.7884035288938053e+09 2loss/SRqqt3/assignment_loss‚1.4303420848612274 .loss/FRt4/detection_loss‚1.643550098836422 -loss/FBt3/detection_loss‚3.91525074839592 /loss/FBt4/detection_loss‚0.6518366414054138 &loss/total_loss‚43.522665261252996 1loss/SRqqt4/assignment_loss‚1.354161686905836 1loss/FBt3/assignment_loss‚0.24793876514861304 /loss/FRt2/detection_loss‚2.8087154369096496 _runtime‚8153.530546567 0loss/SRqqt3/detection_loss‚1.929755019822291 0loss/FRt4/assignment_loss‚1.5140435898303986 /loss/FRt1/detection_loss‚2.0013557220735247 /loss/FRt3/detection_loss‚2.5846097618341446 .loss/FBt1/detection_loss‚2.076553149656816 epoch‚8 _step‚855×≇\»&‚(:’ ϾãÔþç·® gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚142.932' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.586399999999994 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 ŠS¨›¼&‚(:  ϾãÔ“à±$ memory_percent‚96.83716297149658" proc.memory.rssMB‚ 2694.4921875) proc.memory.percent‚16.44587516784668 proc.cpu.threads‚17 disk.nvme0n1.in‚ 404.23046875 disk.sda.out‚ 8061.00390625 network.recv‚ 2.917222e+06 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.dm-0.in‚ 404.23046875 disk.sda.in‚ 0.06640625 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚ 0.0546875* disk./.usagePercent‚15.760622807346673 disk.sda3.in‚ 0.06640625( disk.nvme0c0n1.out‚36879.46630859375 disk.sda2.in‚0% disk./.usageGB‚234.84466934204102 network.sent‚ 2.5760955e+07 disk.sda1.in‚0 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 404.23046875 disk.dm-1.out‚ 8061.00390625 disk.rbd0.out‚ 0.88232421875( proc.memory.availableMB‚ 518.19921875# disk.dm-0.out‚36879.46630859375 disk.sda3.out‚ 8061.00390625 disk.rbd1.in‚0 disk.rbd1.out‚ 21.4755859375 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2265.out‚0 cpu‚25.359129646753484& disk.nvme0n1.out‚36879.46630859375 disk.sda2.out‚0 disk.dm-1.in‚ 0.06640625}‚œ½&‚(:‘ Þ¾ãÔóò“® gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚146.59' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.317999999999998 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0jÊ‚^Š¾&‚(:ÿ Þ¾ãÔò©¢° disk.drbd2080.out‚0 disk.sda3.in‚ 0.06640625( disk.nvme0c0n1.out‚36879.99951171875 disk.dm-1.in‚ 0.06640625 disk.dm-1.out‚ 8061.6953125 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.nvme0c0n1.in‚404.3125 disk.nvme0n1.in‚404.3125 cpu‚25.358570511377383 network.sent‚ 2.5868798e+07 network.recv‚ 2.92264e+06$ disk./.usageGB‚236.6019744873047 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.rbd1.in‚0 disk.rbd1.out‚ 21.4755859375 disk.drbd1716.out‚0 disk.rbd0.in‚ 0.0546875 disk.rbd0.out‚ 0.94482421875# disk.dm-0.out‚36879.99951171875 disk.sda3.out‚ 8061.6953125 disk.sda.out‚ 8061.6953125 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.sda1.out‚0$ memory_percent‚96.83115482330322* disk./.usagePercent‚15.878557030122542 disk.sda.in‚ 0.06640625 disk.drbd2266.out‚0& disk.nvme0n1.out‚36879.99951171875( proc.memory.availableMB‚ 519.18359375" proc.memory.rssMB‚ 2694.4921875) proc.memory.percent‚16.44587516784668 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.dm-0.in‚404.3125 proc.cpu.threads‚17;ýLp2¿&- _wandbruntime‚8188  _runtime‚8188>ÃÿÊÀ&Ä _runtime‚8188.317364692 _step‚856 ' lr-AdamW/pg1‚1.1841077666842139e-05 ' lr-AdamW/pg2‚1.1841077666842139e-05 trainer/global_step‚10449 % _timestamp‚1.7884035636805167e+09ÄÇÏÏÁ&É ' lr-AdamW/pg1‚1.1841077666842139e-05 ' lr-AdamW/pg2‚1.1841077666842139e-05 trainer/global_step‚10449 % _timestamp‚1.7884035636805167e+09 _runtime‚8188.317364692 _step‚856ØHÝs2Â&- _wandbruntime‚8188  _runtime‚8188%:¿1í'Ã&çO XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14736672066560 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14694371121220 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 211231105 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14694205999816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 % _timestamp‚1.7884035636812217e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 _step‚857 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚294485578337280 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14736841066496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 trainer/global_step‚10449 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚309222232400896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚309222419403776 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 211231105 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59231212 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚294485560334336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137694165 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139604533 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚280138498382848 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 348905230 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2ç$´IÄ&ìO VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚309222419403776 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚312031719026688 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚294485560334336 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 137694163 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 211211067 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚294832886543428 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 348925270 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 137694165 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218973184 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚14992616884224 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚309222232400896 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚309222419403776 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 211231105 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59231212 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚294485560334336 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 137694165 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 139604533 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚280138498382848 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 348905230 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 [HDeviceStatsMonitor.on_train_žŽ‹© batch_start/allocated_bytes.small_pool.freed‚14736672066560 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14694371121220 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 211231105 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14694205999816 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 % _timestamp‚1.7884035636812217e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 _step‚857 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚294485578337280 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14736841066496 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 trainer/global_step‚10449 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 348957742 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139617169 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚309250894903808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137706824 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚327054880563712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 _step‚858 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚294860036729544 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137706822 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚280164458604544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 348937676 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884035643550558e+09 _runtime‚8188.992431207 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14738234217984 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14994033823232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚309251081920000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137706824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚294512829699072 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80381233 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14738065204736 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139617683 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚10449 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 348937676 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚280164441565184 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14695595164360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚294512847702016 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 211230854 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969F^Aý&Ç&÷M I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚312060608034816 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 348957742 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 137706822 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14738065204736 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 348957742 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 139617169 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚309250894903808 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 137706824 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚327054880563712 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 _step‚858 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚294860036729544 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 137706822 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚280164458604544 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 348937676 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 % _timestamp‚1.7884035643550558e+09 _runtime‚8188.992431207 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14738234217984 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚14994033823232 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚309251081920000 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 137706824 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚294512829699072 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80381233 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14738065204736 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 139617683 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 trainer/global_step‚10449 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 348937676 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚280164441565184 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14695595164360 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚294512847702016 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚514 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 211230854 PDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚281435674539008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 _runtime‚8224.212322909 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14806512389120 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚295849028571136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80742704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 212223807 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚295849046574080 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 212223807 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14806343389184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚310655371960320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚281435657499648 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 trainer/global_step‚10499 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 140245179 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14806512389120 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚310655371960320 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 140245721 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138327113 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15063791176704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚296199517478636 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚313477342497792 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 _step‚861 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404"# Åó'Ö&íO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138327115 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚313477366437888 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14763677818224 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14806343389184 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚310655371960320 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚281435657499648 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 trainer/global_step‚10499 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 140245179 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚14806512389120 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚310655371960320 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 140245721 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138327113 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15063791176704 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚296199517478636 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚313477342497792 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 _step‚861 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14763842939628 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 350530882 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138327113 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 350550922 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚295849046574080 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚295849028571136 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚80742706 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59503015 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 212203769 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚296199335317872 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚310655558963200 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚310655558963200 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138327115 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚281435674539008 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚542 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 _runtime‚8224.212322909 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14806512389120 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚295849028571136 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚80742704 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 212223807 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚295849046574080 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 212223807 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚296226667664752 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 350563328 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚313506279386112 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚296226849825724 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138339774 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 212243620 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 350583394 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚310684221479424 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚310684034463232 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14807736527360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138339774 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140257928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚281461617721344 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚281461600681984 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14765232104380 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _step‚862 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 212223556 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138339772 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138339772 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 350583394 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14765066982768 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚310684221479424 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚328571490255360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 212243620 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80750083 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59507847 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚328571255744000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 $ _timestamp‚1.788403600253373e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 trainer/global_step‚10499 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15065000297984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚295876297935872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192=¼™Z` Ù&öM XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚296226667664752 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 350563328 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚313506279386112 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚296226849825724 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138339774 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 212243620 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 350583394 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚310684221479424 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚310684034463232 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14807736527360 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 138339774 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140257928 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚281461617721344 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚281461600681984 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_freñÛiyœe‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14765232104380 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 _step‚862 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 212223556 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138339772 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 138339772 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 350583394 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14765066982768 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚310684221479424 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚328571490255360 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 212243620 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚80750083 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59507847 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚328571255744000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 $ _timestamp‚1.788403600253373e+09 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 trainer/global_step‚10499 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15065000297984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚295876297935872 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 350563328 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14807905540608 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 140258450 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15065210869248 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚80750081Þs#â2Ú&- _wandbruntime‚8224  _runtime‚82248W-_Û&Ù 0loss/SRqqt1/detection_loss‚1.713683857562694 $ _timestamp‚1.788403600255107e+09 _step‚863 0loss/FRt1/assignment_loss‚3.2591971099376678 0loss/FBt1/assignment_loss‚0.4625622251944151 /loss/FRt4/detection_loss‚1.7668396847491914 /loss/FBt4/detection_loss‚0.6724606811068952 0loss/FRt4/assignment_loss‚1.3566260186108676 /loss/FBt1/detection_loss‚1.8297671765089034 /loss/FBt3/detection_loss‚1.9748333758778043 1loss/FBt2/assignment_loss‚0.27120792275782907 1loss/SRqqt3/detection_loss‚1.9310278797691518 &loss/total_loss‚41.817171671509456 epoch‚8 2loss/SRqqt4/assignment_loss‚1.1624881844875998 trainer/global_step‚10499 _runtime‚8224.891834605 0loss/FRt3/assignment_loss‚3.0070892336756683 /loss/FRt2/detection_loss‚2.7735645329510725 2loss/SRqqt2/assignment_loss‚1.8891326035890315 1loss/SRqqt4/detection_loss‚1.3206403349510958 /loss/FRt2/assignment_loss‚3.825214739199038 2loss/SRqqt1/assignment_loss‚1.4094841349671812 2loss/SRqqt3/assignment_loss‚1.3171306017532267 0loss/FBt3/assignment_loss‚0.3061681082989607 1loss/FBt4/assignment_loss‚0.14454438207362372 1loss/SRqqt2/detection_loss‚2.4229978306425943 /loss/FBt2/detection_loss‚2.6673842800988092 .loss/FRt1/detection_loss‚2.024432769119774 /loss/FRt3/detection_loss‚2.3086940036263575èÞL¿dÜ&Þ 1loss/FBt2/assignment_loss‚0.27120792275782907 1loss/SRqqt3/detection_loss‚1.9310278797691518 &loss/total_loss‚41.817171671509456 epoch‚8 2loss/SRqqt4/assignment_loss‚1.1624881844875998 trainer/global_step‚10499 _runtime‚8224.891834605 0loss/FRt3/assignment_loss‚3.0070892336756683 /loss/FRt2/detection_loss‚2.7735645329510725 2loss/SRqqt2/assignment_loss‚1.8891326035890315 1loss/SRqqt4/detection_loss‚1.3206403349510958 /loss/FRt2/assignment_loss‚3.825214739199038 2loss/SRqqt1/assignment_loss‚1.4094841349671812 2loss/SRqqt3/assignment_loss‚1.3171306017532267 0loss/FBt3/assignment_loss‚0.3061681082989607 1loss/FBt4/assignment_loss‚0.14454438207362372 1loss/SRqqt2/detection_loss‚2.4229978306425943 /loss/FBt2/detection_loss‚2.6673842800988092 .loss/FRt1/detection_loss‚2.024432769119774 /loss/FRt3/detection_loss‚2.3086940036263575 0loss/SRqqt1/detection_loss‚1.713683857562694 $ _timestamp‚1.788403600255107e+09 _step‚863 0loss/FRt1/assignment_loss‚3.2591971099376678 0loss/FBt1/assignment_loss‚0.4625622251944151 /loss/FRt4/detection_loss‚1.7668396847491914 /loss/FBt4/detection_loss‚0.6724606811068952 0loss/FRt4/assignment_loss‚1.3566260186108676 /loss/FBt1/detection_loss‚1.8297671765089034 /loss/FBt3/detection_loss‚1.9748333758778043ßîÅàÏ’Ý&‚(:‡ š¿ãÔÄŠŠ® gpu.0.gpu‚43.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚118.907' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.7814 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0UŽÛqÞ&‚(:ö š¿ãÔ¥þÔ° disk.nvme0c0n1.in‚ 416.734375 disk.rbd1.out‚ 21.6640625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 416.734375# disk.nvme0n1.out‚40961.44140625 disk.drbd2080.in‚0) proc.memory.percent‚16.44587516784668 disk.sda2.out‚0 disk.dm-1.in‚ 0.06640625 disk.rbd0.in‚ 0.0546875 disk.rbd1.in‚0' proc.memory.availableMB‚ 518.9765625 network.sent‚ 2.5995181e+07% disk.nvme0c0n1.out‚40961.44140625 disk.sda.out‚ 8064.2109375 disk.dm-1.out‚ 8064.2109375 disk.drbd2080.out‚0" proc.memory.rssMB‚ 2694.4921875) disk./.usagePercent‚16.02752986235724 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 8064.2109375 disk.drbd2266.in‚0$ memory_percent‚96.83241844177246 network.recv‚ 2.941348e+06" disk./.usageGB‚238.82177734375 disk.drbd1716.in‚0 disk.drbd2266.out‚0 cpu‚25.356296259961276 disk.rbd0.out‚ 1.03564453125 disk.dm-0.in‚ 416.734375 disk.dm-0.out‚40961.44140625 disk.sda1.in‚0 disk.sda.in‚ 0.06640625 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚ 0.06640625O?–ß&‚(:’ ©¿ãÔæ·˜® gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚120.558' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.111600000000003 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0º.K à&‚(:‚  ©¿ãÔ—±í° disk.rbd0.out‚ 1.03564453125 disk.rbd1.out‚ 21.8203125( disk.nvme0c0n1.out‚40961.87451171875" proc.memory.rssMB‚ 2694.4921875 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.nvme0c0n1.in‚ 416.734375 network.sent‚ 2.6102768e+07 disk.sda.out‚ 8064.75390625# disk.dm-0.out‚40961.87451171875 disk.rbd1.in‚0 disk.sda1.out‚0 network.recv‚ 2.946678e+06 disk.dm-1.in‚ 0.06640625 disk.drbd2080.out‚0 disk.dm-0.in‚ 416.734375( proc.memory.availableMB‚ 518.96484375) proc.memory.percent‚16.44587516784668& disk.nvme0n1.out‚40961.87451171875 disk.sda.in‚ 0.06640625 disk.drbd2265.in‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.06640625" disk./.usageGB‚238.82177734375$ memory_percent‚96.83248996734619 cpu‚25.35567940463749 proc.cpu.threads‚17 disk.nvme0n1.in‚ 416.734375 disk.drbd1716.in‚0 disk.rbd0.in‚ 0.0546875 disk.sda3.out‚ 8064.75390625) disk./.usagePercent‚16.02752986235724 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 8064.75390625 disk.drbd2266.in‚0 disk.drbd2265.out‚0 µ862á&- _wandbruntime‚8260  _runtime‚8260ðN«Éâ&à ' lr-AdamW/pg2‚1.0465084240557901e-05 trainer/global_step‚10549 $ _timestamp‚1.788403635396592e+09 _runtime‚8260.033455136 _step‚864 ' lr-AdamW/pg1‚1.0465084240557901e-05Ûþc“Îã&È _runtime‚8260.033455136 _step‚864 ' lr-AdamW/pg1‚1.0465084240557901e-05 ' lr-AdamW/pg2‚1.0465084240557901e-05 trainer/global_step‚10549 $ _timestamp‚1.788403635396592e+09à¼,“é2ä&- _wandbruntime‚8260  _runtime‚8260ËMÛí'å&çO TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81111554 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚297212514810880 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚312088698522624 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 213196471 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚14876183711744 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚312088698522624 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138960065 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 213196471 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚314922989908992 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 213216509 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚865 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚312088511519744 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚297565966253080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚297212496807936 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15134767839232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 140885416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14833149636632 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚330057547163648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 352156534 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138960065 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59773860 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 213216509 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚314923013849088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14876014711808 trainer/global_step‚10549 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14876014711808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 352176574 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚330057781688320 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15134557254656 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚282732816616448 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138960063 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 140884871 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59773317 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138960063 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚312088511519744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚312088511519744 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚543 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚312088698522624 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 138960065 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 213196471 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚314922989908992 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 213216509 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚865 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚312088511519744 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚297565966253080 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚297212496807936 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15134767839232 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 140885416 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14833149636632 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚330057547163648 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 352156534 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 138960065 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚59773860 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 213216509 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚314923013849088 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚14876014711808 trainer/global_step‚10549 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚14876014711808 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 352176574 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚330057781688320 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15134557254656 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚282732816616448 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 138960063 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 140884871 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚59773317 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 SDDeviceStatsMonitor.on_trainnQ*r _batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 138960063 JDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 352209046 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884036360638206e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚312117361038848 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14834538801176 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 213216258 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81118931 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚494 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138972724 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚297593298599960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚866 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14877576863232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚330087887208448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213216258 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14877407849984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81118933 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚314951926797312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14834703922788 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚8260.701226326 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202182656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚312117174022656 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15135984351232 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚297239766172672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚496 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140897447 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 213236322 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128¤¬üý&é&÷M GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 trainer/global_step‚10549 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 _runtime‚8260.701226326 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202182656 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚312117174022656 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15135984351232 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚297239766172672 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚496 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 140897447 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 213236322 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 352188980 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚282758776838144 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 352209046 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚312117361038848 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚297239784175616 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14877576863232 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 138972722 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚59778516 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 352209046 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884036360638206e+09 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚312117361038848 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14834538801176 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 213216258 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81118931 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚494 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 138972724 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚297593298599960 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _step‚866 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14877576863232 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚330087887208448 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 213216258 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14877407849984 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81118933 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚314951926797312 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14834703922788 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 EŒî&‚(:  ¸¿ãÔìê¿° network.sent‚ 2.6108812e+07 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.rbd0.out‚ 1.0986328125 disk.rbd1.in‚0 disk.sda.in‚ 0.06640625 disk.sda2.in‚0 network.recv‚ 2.950901e+06) disk./.usagePercent‚15.93864327844641 disk.rbd0.in‚ 0.0546875 disk.sda3.out‚ 8065.4453125 disk.dm-0.in‚ 416.734375 disk.sda3.in‚ 0.06640625 disk.sda.out‚ 8065.4453125 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.06640625( proc.memory.availableMB‚ 519.13671875" proc.memory.rssMB‚ 2694.4921875 cpu‚25.355128556748692 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.nvme0n1.in‚ 416.734375& disk.nvme0n1.out‚40962.93603515625$ disk./.usageGB‚237.4973030090332 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 416.734375( disk.nvme0c0n1.out‚40962.93603515625 disk.drbd1716.out‚0 disk.dm-1.out‚ 8065.4453125# disk.dm-0.out‚40962.93603515625 disk.sda1.in‚0 disk.rbd1.out‚ 21.8203125$ memory_percent‚96.83144092559814) proc.memory.percent‚16.44587516784668‡AQï’ï&‚(:‡ Ç¿ãÔœ·ü­ gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚123.751' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.7502 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03ì¿Šð&‚(:ÿ Ç¿ãÔÏ©Å° disk.dm-0.in‚ 416.734375 disk.sda1.out‚0' disk.nvme0c0n1.out‚40964.5732421875 disk.nvme0n1.in‚ 416.734375 disk.sda.out‚ 8066.2734375 network.sent‚ 2.621764e+07$ disk./.usageGB‚237.4972686767578 disk.drbd2080.in‚0 disk.rbd0.out‚ 1.0986328125 disk.sda3.out‚ 8066.2734375 disk.rbd1.out‚ 21.8525390625 disk.drbd2266.out‚0 disk.sda1.in‚0 disk.sda.in‚ 0.06640625$ memory_percent‚96.83535099029541 disk.sda2.out‚0 disk.rbd0.in‚ 0.0546875( proc.memory.availableMB‚ 518.49609375 disk.sda2.in‚0 disk.drbd2265.out‚0) proc.memory.percent‚16.44587516784668 disk.drbd2266.in‚0 network.recv‚ 2.957261e+06( disk./.usagePercent‚15.9386409743786 disk.rbd1.in‚0% disk.nvme0n1.out‚40964.5732421875 cpu‚25.354577248891403" disk.dm-0.out‚40964.5732421875 disk.sda3.in‚ 0.06640625" proc.memory.rssMB‚ 2694.4921875 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.06640625 disk.nvme0c0n1.in‚ 416.734375 proc.cpu.threads‚17 disk.dm-1.out‚ 8066.2734375 disk.drbd2265.in‚0 disk.drbd2080.out‚06L‘ñ&‚(:’ Ö¿ãÔܧè® gpu.0.gpu‚19.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚119.444' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚23.888800000000003 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0;xþ‹ò&‚(:€  Ö¿ãÔÛ´€° disk.sda2.out‚0 disk.drbd1716.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.06640625 disk.nvme0c0n1.in‚ 416.734375% disk.nvme0n1.out‚40964.8037109375) proc.memory.percent‚16.44587516784668 disk.sda.out‚ 8066.8671875 disk.drbd2080.out‚0 disk.rbd1.out‚ 21.8525390625' disk.nvme0c0n1.out‚40964.8037109375 proc.cpu.threads‚17 disk.drbd2265.out‚0 disk.sda3.out‚ 8066.8671875 disk.rbd1.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.nvme0n1.in‚ 416.734375" proc.memory.rssMB‚ 2694.4921875 network.sent‚ 2.6223014e+07 network.recv‚ 2.960936e+06$ disk./.usageGB‚237.4972686767578 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.06640625 disk.dm-1.out‚ 8066.8671875 disk.rbd0.in‚ 0.0546875 disk.dm-0.in‚ 416.734375$ memory_percent‚96.83220386505127 cpu‚25.354031492172332( disk./.usagePercent‚15.9386409743786 disk.sda.in‚ 0.06640625 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0" disk.dm-0.out‚40964.8037109375( proc.memory.availableMB‚ 519.01171875 disk.sda2.in‚0 disk.rbd0.out‚ 1.0986328125K†+û2ó&- _wandbruntime‚8295  _runtime‚8295äóFÇô&Á $ _timestamp‚1.788403670956584e+09 _runtime‚8295.593463903 _step‚868 & lr-AdamW/pg1‚9.807347689029617e-06 & lr-AdamW/pg2‚9.807347689029617e-06 trainer/global_step‚10599Ið*ÁÌõ&Æ trainer/global_step‚10599 $ _timestamp‚1.788403670956584e+09 _runtime‚8295.593463903 _step‚868 & lr-AdamW/pg1‚9.807347689029617e-06 & lr-AdamW/pg2‚9.807347689029617e-06ä×€$2ö&- _wandbruntime‚8295  _runtime‚8295¬E þ÷&èO QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139593015 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 214209211 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚313521838082048 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139593013 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚553 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 353782186 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚313521651079168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚316368661260288 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚331574397429248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚298932779349052 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚313521651079168 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15205521390080 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚313521838082048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 353802226 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14902621455040 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚298575965044736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81480406 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 _runtime‚8295.594404377 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 214209211 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 214189173 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚284029992772608 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 141525081 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚10599;0•ó'ø&íO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 214209211 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 214189173 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚284029992772608 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 141525081 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 trainer/global_step‚10599 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 139593015 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 214209211 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚313521838082048 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 139593013 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚553 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 353782186 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚313521651079168 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚316368661260288 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚331574397429248 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚298932779349052 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚313521651079168 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15205521390080 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚313521838082048 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 353802226 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14902621455040 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚298575965044736 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81480406 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 _runtime‚8295.594404377 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064åB¯­©2ù&- _wandbruntime‚8296  _runtime‚8296ѹ°æø&ú&òM ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14947248185856 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15207152455680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚298959929535168 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚10599 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14947079172608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚284055918915584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚555 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 % _timestamp‚1.7884036716260688e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚298603252412416 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚331604726664192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 353834698 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14947079172608 _runtime‚8296.263490143 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚313550500598272 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 353814632 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139605674 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚331604490055680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 139605672 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141537242 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚316397574208512 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139605674 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 214229024 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚298603234409472 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141537797 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚553 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚298960111696140 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 214229024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15206939787264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 214208960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚284055935954944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81487783 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚298603234409472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚316397550268416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14904175741196 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14904010619584 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚313550313582080 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81487781 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60049461 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚870 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14947248185856 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 353834698 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760ÿÿ«Ðû&÷M [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚298603252412416 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚331604726664192 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 353834698 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚14947079172608 _runtime‚8296.263490143 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚313550500598272 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 353814632 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 139605674 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚331604490055680 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 139605672 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 141537242 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚316397574208512 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 139605674 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 214229024 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚298603234409472 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 141537797 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚553 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚298960111696140 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 214229024 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15206939787264 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 214208960 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚284055935954944 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81487783 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚298603234409472 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚316397550268416 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14904175741196 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚14904010619584 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚313550313582080 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚81487781 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60049461 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 BDeviceStatsMonitor.on_train_batch_end/segmentî®]‚-.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 236608512 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 _step‚870 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚14947248185856 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 353834698 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚14947248185856 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15207152455680 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚298959929535168 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚10599 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚14947079172608 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚284055918915584 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚555 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 % _timestamp‚1.7884036716260688e+09 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401æÞ6v2ü&- _wandbruntime‚8296  _runtime‚8296ÐTJVý&Ð 2loss/SRqqt1/assignment_loss‚1.4665914554843458 /loss/FBt2/assignment_loss‚0.929006167454645 1loss/SRqqt4/detection_loss‚0.9818015391366524 .loss/FRt3/detection_loss‚2.361456587528571 /loss/FBt2/detection_loss‚3.3445290525754294 epoch‚8 trainer/global_step‚10599 _runtime‚8296.264657194 /loss/FRt2/assignment_loss‚3.936079440208582 2loss/SRqqt4/assignment_loss‚0.9243255524912424 1loss/SRqqt1/detection_loss‚1.5313580571594885 %loss/total_loss‚44.34092015386957 % _timestamp‚1.7884036716278985e+09 0loss/FBt3/assignment_loss‚0.3806403875349978 .loss/FRt4/detection_loss‚1.818125059099301 .loss/FBt3/detection_loss‚3.305884974343436 /loss/FRt4/assignment_loss‚1.536685716846715 1loss/SRqqt2/assignment_loss‚1.774261939738478 1loss/FBt4/assignment_loss‚0.11574303894368035 /loss/FRt2/detection_loss‚3.1240225846950826 1loss/SRqqt2/detection_loss‚2.4184864440134595 /loss/FBt4/detection_loss‚0.7094269326370624 0loss/FRt1/assignment_loss‚3.2834117293357847 1loss/SRqqt3/assignment_loss‚1.467636997335487 _step‚871 0loss/FRt3/assignment_loss‚2.6269136865933738 0loss/FBt1/assignment_loss‚0.2937516926590783 .loss/FRt1/detection_loss‚2.064569524526596 0loss/SRqqt3/detection_loss‚1.803636136982176 .loss/FBt1/detection_loss‚2.142575456545903>:[ñ[þ&Õ 2loss/SRqqt4/assignment_loss‚0.9243255524912424 1loss/SRqqt1/detection_loss‚1.5313580571594885 %loss/total_loss‚44.34092015386957 % _timestamp‚1.7884036716278985e+09 0loss/FBt3/assignment_loss‚0.3806403875349978 .loss/FRt4/detection_loss‚1.818125059099301 .loss/FBt3/detection_loss‚3.305884974343436 /loss/FRt4/assignment_loss‚1.536685716846715 1loss/SRqqt2/assignment_loss‚1.774261939738478 1loss/FBt4/assignment_loss‚0.11574303894368035 /loss/FRt2/detection_loss‚3.1240225846950826 1loss/SRqqt2/detection_loss‚2.4184864440134595 /loss/FBt4/detection_loss‚0.7094269326370624 0loss/FRt1/assignment_loss‚3.2834117293357847 1loss/SRqqt3/assignment_loss‚1.467636997335487 _step‚871 0loss/FRt3/assignment_loss‚2.6269136865933738 0loss/FBt1/assignment_loss‚0.2937516926590783 .loss/FRt1/detection_loss‚2.064569524526596 0loss/SRqqt3/detection_loss‚1.803636136982176 .loss/FBt1/detection_loss‚2.142575456545903 2loss/SRqqt1/assignment_loss‚1.4665914554843458 /loss/FBt2/assignment_loss‚0.929006167454645 1loss/SRqqt4/detection_loss‚0.9818015391366524 .loss/FRt3/detection_loss‚2.361456587528571 /loss/FBt2/detection_loss‚3.3445290525754294 epoch‚8 trainer/global_step‚10599 _runtime‚8296.264657194 /loss/FRt2/assignment_loss‚3.936079440208582ç>f,2ÿ&‚(:’ å¿ãÔ䇜® gpu.0.gpu‚29.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚121.088' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.217599999999997 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0îæ–‰€'‚(:þ å¿ãÔκû° disk.drbd1716.in‚0 disk.drbd2265.out‚0# disk.dm-0.out‚40966.54931640625 disk.sda3.out‚ 8067.515625 disk.rbd1.out‚ 22.0087890625 proc.cpu.threads‚17 disk.nvme0n1.in‚ 416.734375 disk.sda.in‚ 0.06640625 disk.dm-1.in‚ 0.06640625 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0) proc.memory.percent‚16.44587516784668 cpu‚25.3535082408867 network.recv‚ 2.965311e+06) disk./.usagePercent‚15.93864148639367 disk.sda3.in‚ 0.06640625& disk.nvme0n1.out‚40966.54931640625 disk.sda.out‚ 8067.515625 disk.sda2.in‚0 disk.rbd1.in‚0% disk./.usageGB‚237.49727630615234 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.rbd0.out‚ 1.0986328125# memory_percent‚96.8339204788208( proc.memory.availableMB‚ 518.73046875" proc.memory.rssMB‚ 2694.4921875 disk.drbd2265.in‚0 disk.rbd0.in‚ 0.0546875 disk.dm-0.in‚ 416.734375( disk.nvme0c0n1.out‚40966.54931640625 disk.dm-1.out‚ 8067.515625 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 416.734375 network.sent‚ 2.622903e+07 disk.sda2.out‚0LYMÙ'‚(:’ ô¿ãÔƒû®® gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚132.419' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.483800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0x%°‰‚'‚(:þ ô¿ãÔÀ¬‡° disk.sda.out‚ 8068.00390625 disk.sda2.in‚0 disk.dm-1.in‚ 0.06640625 disk.rbd1.out‚ 22.0087890625$ disk.nvme0n1.out‚40966.869140625" proc.memory.rssMB‚ 2694.4921875 disk.dm-1.out‚ 8068.00390625 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.sda1.in‚0& disk.nvme0c0n1.out‚40966.869140625% proc.memory.availableMB‚ 518.96875 network.recv‚ 2.971302e+06* disk./.usagePercent‚15.938647118559423 disk.sda2.out‚0 disk.drbd2080.in‚0 proc.cpu.threads‚17 network.sent‚ 2.633756e+07 cpu‚25.352938857142288 disk.rbd0.in‚ 0.0546875 disk.sda1.out‚0 disk.sda3.out‚ 8068.00390625 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 416.734375! disk.dm-0.out‚40966.869140625$ disk./.usageGB‚237.4973602294922 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.sda3.in‚ 0.06640625 disk.nvme0c0n1.in‚ 416.734375$ memory_percent‚96.83246612548828) proc.memory.percent‚16.44587516784668 disk.rbd0.out‚ 1.0986328125 disk.rbd1.in‚0 disk.nvme0n1.in‚ 416.734375 disk.sda.in‚ 0.06640625P²ôß2ƒ'- _wandbruntime‚8331  _runtime‚8331 A_ÌÇ„'Á _step‚872 & lr-AdamW/pg1‚9.169968744969032e-06 & lr-AdamW/pg2‚9.169968744969032e-06 trainer/global_step‚10649 % _timestamp‚1.7884037063917606e+09 _runtime‚ 8331.028613282,yÌ…'Æ & lr-AdamW/pg1‚9.169968744969032e-06 & lr-AdamW/pg2‚9.169968744969032e-06 trainer/global_step‚10649 % _timestamp‚1.7884037063917606e+09 _runtime‚ 8331.02861328 _step‚872èÌ+_2†'- _wandbruntime‚8331  _runtime‚8331Ô9‡yî'‡'èO \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15015526356992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚299939451284480 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚81849256 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 _step‚873 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 355427878 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚314954790638592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 215201913 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60317227 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚300299228123496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 355407838 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60317767 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚300299410284260 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 215181875 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚299939433281536 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚299939451284480 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 215181875 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚285327134850048 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚314954977641472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚81849254 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚14972258394852 trainer/global_step‚10649 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚299939433281536 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15015526356992 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15276399228928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140225963 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15276611910656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚333090683960320 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚317814284731392 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚333090920582144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚314954977641472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140225965 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884037063924081e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140225963 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15015357357056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14972093273448 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚299939433281536 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15015526356992 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15276399228928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140225963 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15276611910656 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚333090683960320 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚317814284731392 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚333090920582144 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚314954977641472 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 140225965 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚540 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 % _timestamp‚1.7884037063924081e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140225963 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15015357357056 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚14972093273448 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 _step‚873 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 355427878 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚314954790638592 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 215201913 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60317227 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚300299228123496 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 355407838 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚60317767 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚300299410284260 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 215181875 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚299939433281536 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408é)‡Ò»2‰'- _wandbruntime‚8331  _runtime‚8331‡ê—*Š'ñM E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚299966702646272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚314983640157696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 215221726 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚285353078032384 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140238622 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚300326560470376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 355460350 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 215201662 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60323021 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚300326742631348 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15016919508480 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 215221726 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 355440284 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15277819040256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚333121016719872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15278029611520 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚299966702646272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15016919508480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 % _timestamp‚1.7884037070672636e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚299966720649216 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚317843221619712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60322498 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81856633 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14973647559604 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142179129 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140238622 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 8331.70453251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15016750495232 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140238624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚314983640157696 trainer/global_step‚10649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 355440284 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚285353095071744 _step‚874 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 142179654 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985{^ûü&‹'öM ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚285353095071744 _step‚874 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 142179654 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚299966702646272 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚314983640157696 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 215221726 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚285353078032384 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140238622 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚300326560470376 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 355460350 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 215201662 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60323021 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚300326742631348 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15016919508480 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 215221726 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 355440284 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15277819040256 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚333121016719872 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15278029611520 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚299966702646272 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15016919508480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 % _timestamp‚1.7884037070672636e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚299966720649216 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚317843221619712 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚523 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚60322498 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚81856633 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚14973647559604 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142179129 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140238622 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚ 8331.70453251 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15016750495232 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 140238624 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚525 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚314983640157696 trainer/global_step‚10649 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 355440284 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416êµyd2Œ'- _wandbruntime‚8331  _runtime‚8331¡å¿\'Ö 0loss/FRt3/assignment_loss‚2.8396748798649485 /loss/FRt2/detection_loss‚2.8958597199784384 1loss/SRqqt1/detection_loss‚1.6442386054744322 0loss/SRqqt4/detection_loss‚1.035296049994835 /loss/FRt1/detection_loss‚1.8620801232755355 1loss/SRqqt2/detection_loss‚2.4976706342263655 2loss/SRqqt4/assignment_loss‚1.0171361454371093 /loss/FRt4/detection_loss‚1.6706531106361322 trainer/global_step‚10649 0loss/FRt4/assignment_loss‚1.0565615104777473 1loss/SRqqt1/assignment_loss‚1.512479176162742 1loss/SRqqt3/assignment_loss‚1.377133746982886 .loss/FRt3/detection_loss‚2.358288397149342 _step‚875 0loss/FRt1/assignment_loss‚2.5660688681527972 /loss/FRt2/assignment_loss‚3.854059351815118 1loss/FBt1/assignment_loss‚0.23303654352397038 1loss/SRqqt3/detection_loss‚1.6821691694686998 .loss/FBt3/detection_loss‚3.000147432088852 2loss/SRqqt2/assignment_loss‚1.6762321866822965 1loss/FBt2/assignment_loss‚0.29844206614500135 0loss/FBt4/assignment_loss‚0.1891583149811756 /loss/FBt4/detection_loss‚0.6037684454414476 %loss/total_loss‚40.35646640115978 _runtime‚8331.705611475 2loss/FBt3/assignment_loss‚0.058490506314483603 epoch‚8 .loss/FBt1/detection_loss‚1.929335484902064 /loss/FBt2/detection_loss‚2.4984859319833608 $ _timestamp‚1.788403707068878e+09ñšq]aŽ'Û 0loss/FRt4/assignment_loss‚1.0565615104777473 1loss/SRqqt1/assignment_loss‚1.512479176162742 1loss/SRqqt3/assignment_loss‚1.377133746982886 .loss/FRt3/detection_loss‚2.358288397149342 _step‚875 0loss/FRt1/assignment_loss‚2.5660688681527972 /loss/FRt2/assignment_loss‚3.854059351815118 1loss/FBt1/assignment_loss‚0.23303654352397038 1loss/SRqqt3/detection_loss‚1.6821691694686998 .loss/FBt3/detection_loss‚3.000147432088852 2loss/SRqqt2/assignment_loss‚1.6762321866822965 1loss/FBt2/assignment_loss‚0.29844206614500135 0loss/FBt4/assignment_loss‚0.1891583149811756 /loss/FBt4/detection_loss‚0.6037684454414476 %loss/total_loss‚40.35646640115978 _runtime‚8331.705611475 2loss/FBt3/assignment_loss‚0.058490506314483603 epoch‚8 .loss/FBt1/detection_loss‚1.929335484902064 /loss/FBt2/detection_loss‚2.4984859319833608 $ _timestamp‚1.788403707068878e+09 0loss/FRt3/assignment_loss‚2.8396748798649485 /loss/FRt2/detection_loss‚2.8958597199784384 1loss/SRqqt1/detection_loss‚1.6442386054744322 0loss/SRqqt4/detection_loss‚1.035296049994835 /loss/FRt1/detection_loss‚1.8620801232755355 1loss/SRqqt2/detection_loss‚2.4976706342263655 2loss/SRqqt4/assignment_loss‚1.0171361454371093 /loss/FRt4/detection_loss‚1.6706531106361322 trainer/global_step‚10649ë“'éª'‚(:… ƒÀãÔÛƤ® gpu.0.gpu‚36.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚35.0 gpu.0.powerWatts‚140.51' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.102 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0]ÅS¦Ž'‚(:ƒ  ƒÀãÔÐѱ disk.dm-1.out‚ 8068.44921875 disk.sda1.in‚0) proc.memory.percent‚16.44587516784668 network.sent‚ 2.6343864e+07& disk.nvme0n1.out‚40967.71142578125 cpu‚25.35238813550046 proc.cpu.threads‚17 disk.sda.out‚ 8068.44921875 disk.sda2.in‚0 disk.dm-1.in‚ 0.06640625 disk.rbd0.in‚ 0.0546875* disk./.usagePercent‚15.938647118559423 disk.drbd1716.in‚0 disk.sda1.out‚0 disk.sda3.out‚ 8068.44921875 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚ 1.0986328125 disk.sda3.in‚ 0.06640625# memory_percent‚96.8327522277832 disk.sda2.out‚0 disk.dm-0.in‚ 416.734375 disk.rbd1.in‚0( disk.nvme0c0n1.out‚40967.71142578125 disk.nvme0n1.in‚ 416.734375& proc.memory.availableMB‚ 518.921875$ disk./.usageGB‚237.4973602294922 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0# disk.dm-0.out‚40967.71142578125 disk.rbd1.out‚ 22.041015625 disk.nvme0c0n1.in‚ 416.734375" proc.memory.rssMB‚ 2694.4921875 disk.drbd2266.in‚0 disk.drbd2266.out‚0 network.recv‚ 2.975735e+06 disk.sda.in‚ 0.06640625‰ÜÓ,’‘'‚(:‡ ’ÀãÔè³®® gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚125.106' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.0212 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆ"eWƒ’'‚(:ø ’ÀãÔßßа disk.rbd1.in‚0( disk.nvme0c0n1.out‚40967.92236328125' proc.memory.availableMB‚ 519.1484375" proc.memory.rssMB‚ 2694.4921875 disk.dm-0.in‚ 416.734375 disk.sda1.in‚0 network.sent‚ 2.645183e+07 network.recv‚ 2.981262e+06 disk.drbd1716.in‚0 disk.rbd0.in‚ 0.0546875 disk.rbd0.out‚ 1.0986328125 disk.drbd2266.out‚0 disk.sda3.out‚ 8068.9375 disk.rbd1.out‚ 22.041015625 disk.nvme0c0n1.in‚ 416.734375 disk.nvme0n1.in‚ 416.734375 cpu‚25.35185065752446 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.06640625 disk.dm-1.out‚ 8068.9375 disk.drbd2266.in‚0# disk.dm-0.out‚40967.92236328125$ memory_percent‚96.83136940002441* disk./.usagePercent‚15.938647118559423 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.06640625& disk.nvme0n1.out‚40967.92236328125) proc.memory.percent‚16.44587516784668 proc.cpu.threads‚17$ disk./.usageGB‚237.4973602294922 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda.in‚ 0.06640625 disk.sda.out‚ 8068.9375}.?È2“'- _wandbruntime‚8366  _runtime‚8366¯»”žÈ”' % _timestamp‚1.7884037421813667e+09 _runtime‚8366.818224175 _step‚876 & lr-AdamW/pg1‚8.553080237192776e-06 & lr-AdamW/pg2‚8.553080237192776e-06 trainer/global_step‚10699pÛÌÍ•'Ç & lr-AdamW/pg1‚8.553080237192776e-06 & lr-AdamW/pg2‚8.553080237192776e-06 trainer/global_step‚10699 % _timestamp‚1.7884037421813667e+09 _runtime‚8366.818224175 _step‚876ìá·”2–'- _wandbruntime‚8366  _runtime‚8366ôÏôhº—'èO N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 142808161 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15085028679680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚301302919521280 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚301665859058704 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15041730213260 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140858913 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚877 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82218106 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316387930198016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚301666041219468 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 216174577 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 142808729 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 216194615 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 trainer/global_step‚10699 _runtime‚8366.819102381 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚301302919521280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140858913 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚319259956082688 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140858915 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚316388117200896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚286624293966848 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 357053530 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15085028679680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 216174577 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚334607500539392 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚334607259723264 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15347544456704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15347327580672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_s3Ì4tart/segment.small_pool.allocated‚199 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15041565091856 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚316387930198016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60590057 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚286624311006208 % _timestamp‚1.7884037421820736e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15085197679616 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚319259932142592 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217CÁ}Áó'˜'íO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚82218106 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚316387930198016 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚301666041219468 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 216174577 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 142808729 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 216194615 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 trainer/global_step‚10699 _runtime‚8366.819102381 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚301302919521280 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 140858913 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚319259956082688 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 140858915 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚316388117200896 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚286624293966848 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 357053530 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15085028679680 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 216174577 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚334607500539392 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚334607259723264 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15347544456704 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15347327580672 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15041565091856 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚316387930198016 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚60590057 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚286624311006208 % _timestamp‚1.7884037421820736e+09 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15085197679616 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚319259932142592 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 142808161 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15085028679680 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚301302919521280 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚301665859058704 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15041730213260 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 140858913 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _step‚877 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚286650254188544 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 216194364 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚316416779717120 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚301693191405584 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚286650237149184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚301330170883072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15348746900480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 357086002 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140871572 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15086590831104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 % _timestamp‚1.7884037428548286e+09 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15348959568896 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚316416592700928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 357065936 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15042954256400 _runtime‚8367.492230584 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15086590831104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142821248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚319288869030912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 357086002 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 trainer/global_step‚10699 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 216194364 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140871572 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _step‚878 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚316416779717120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚301330170883072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚319288845090816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82225481 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82225483 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚334637591991296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60596310 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 357065936 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15043119378012 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚301330188886016 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 142821793 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15086421817856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140871574 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚301330188886016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 216214428N•É¼ý&›'÷M SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15348746900480 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 357086002 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 140871572 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15086590831104 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 % _timestamp‚1.7884037428548286e+09 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15348959568896 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚316416592700928 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚545 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 357065936 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15042954256400 _runtime‚8367.492230584 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15086590831104 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 142821248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚319288869030912 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 EDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 357086002 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 trainer/global_step‚10699 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 216194364 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 140871572 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _step‚878 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚316416779717120 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚301330170883072 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚319288845090816 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82225481 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82225483 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚334637591991296 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60596310 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚543 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 357065936 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15043119378012 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚301330188886016 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 142821793 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15086421817856 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 140871574 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚301330188886016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 216214428 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 216214428 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚286650254188544 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 216194364 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚316416779717120 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚301693191405584 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚286650237149184 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚301330170883072 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0îH‰Òa2œ'- _wandbruntime‚8367  _runtime‚8367°ºýO'Ó 0loss/FRt4/assignment_loss‚1.0810997092298098 2loss/SRqqt2/assignment_loss‚1.8205019841271062 0loss/FBt1/assignment_loss‚0.2659281581462613 0loss/FBt3/assignment_loss‚0.5738316528731957 _runtime‚ 8367.4933103 2loss/SRqqt3/assignment_loss‚1.3897976133852237 2loss/SRqqt4/assignment_loss‚0.909339198296Úe 8286 1loss/FBt2/assignment_loss‚0.23690933848571988 1loss/FBt4/assignment_loss‚0.17283117184717273 /loss/FRt4/detection_loss‚1.4827006690736328 1loss/SRqqt3/detection_loss‚1.5555770871860366 _step‚879 /loss/FRt1/detection_loss‚2.1160396951801967 0loss/SRqqt4/detection_loss‚1.073989828590487 trainer/global_step‚10699 % _timestamp‚1.7884037428565614e+09 /loss/FRt1/assignment_loss‚3.133107126975546 1loss/SRqqt1/assignment_loss‚1.277031813786986 .loss/FRt2/detection_loss‚3.149479308298656 1loss/SRqqt1/detection_loss‚1.6228212309069932 /loss/FBt1/detection_loss‚1.9203457994894548 0loss/FRt3/assignment_loss‚3.1700359666070272 /loss/FRt3/detection_loss‚2.3687552137430323 0loss/SRqqt2/detection_loss‚2.596499497852018 .loss/FBt2/detection_loss‚2.474648255568284 %loss/total_loss‚41.32339612995756 /loss/FRt2/assignment_loss‚3.332552228655134 /loss/FBt3/detection_loss‚2.9817877262830734 /loss/FBt4/detection_loss‚0.6177858553696857 epoch‚8ŒÀþ”^ž'Ø 0loss/FRt4/assignment_loss‚1.0810997092298098 2loss/SRqqt2/assignment_loss‚1.8205019841271062 0loss/FBt1/assignment_loss‚0.2659281581462613 0loss/FBt3/assignment_loss‚0.5738316528731957 _runtime‚ 8367.4933103 2loss/SRqqt3/assignment_loss‚1.3897976133852237 2loss/SRqqt4/assignment_loss‚0.9093391982968286 1loss/FBt2/assignment_loss‚0.23690933848571988 1loss/FBt4/assignment_loss‚0.17283117184717273 /loss/FRt4/detection_loss‚1.4827006690736328 1loss/SRqqt3/detection_loss‚1.5555770871860366 _step‚879 /loss/FRt1/detection_loss‚2.1160396951801967 0loss/SRqqt4/detection_loss‚1.073989828590487 trainer/global_step‚10699 % _timestamp‚1.7884037428565614e+09 /loss/FRt1/assignment_loss‚3.133107126975546 1loss/SRqqt1/assignment_loss‚1.277031813786986 .loss/FRt2/detection_loss‚3.149479308298656 1loss/SRqqt1/detection_loss‚1.6228212309069932 /loss/FBt1/detection_loss‚1.9203457994894548 0loss/FRt3/assignment_loss‚3.1700359666070272 /loss/FRt3/detection_loss‚2.3687552137430323 0loss/SRqqt2/detection_loss‚2.596499497852018 .loss/FBt2/detection_loss‚2.474648255568284 %loss/total_loss‚41.32339612995756 /loss/FRt2/assignment_loss‚3.332552228655134 /loss/FBt3/detection_loss‚2.9817877262830734 /loss/FBt4/detection_loss‚0.6177858553696857 epoch‚8’Ÿ'‚(:‡ ¡ÀãÔÏ£º® gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚129.179' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.8358 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0z¾Ò '‚(:‚  ¡ÀãÔ—µï° disk.rbd1.out‚ 22.197265625& disk.nvme0n1.out‚40968.25634765625" proc.memory.rssMB‚ 2694.4921875 network.sent‚ 2.645787e+07 network.recv‚ 2.98542e+06 disk.dm-0.in‚ 416.734375( disk.nvme0c0n1.out‚40968.25634765625* disk./.usagePercent‚15.938647118559423 disk.sda2.out‚0 disk.drbd1716.out‚0( proc.memory.availableMB‚ 518.86328125 proc.cpu.threads‚17 disk.sda1.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚ 0.0546875 disk.sda.in‚ 0.06640625 disk.sda.out‚ 8069.5859375 disk.drbd2266.out‚0$ memory_percent‚96.83310985565186$ disk./.usageGB‚237.4973602294922 disk.sda3.in‚ 0.06640625 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 416.734375 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0# disk.dm-0.out‚40968.25634765625 disk.sda1.in‚0) proc.memory.percent‚16.44587516784668 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2265.out‚0 cpu‚25.351309841621614 disk.nvme0n1.in‚ 416.734375 disk.dm-1.out‚ 8069.5859375 disk.rbd0.out‚ 1.0986328125 disk.sda3.out‚ 8069.5859375TR)’¡'‚(:‡ °ÀãÔ㛀® gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚128.642' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.7284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«ÓÜ‹†¢'‚(:û °ÀãÔ¿Ö…° disk.drbd1716.in‚0 disk.dm-1.in‚ 0.06640625 cpu‚25.35077763291523 disk.drbd2265.out‚0 disk.rbd0.in‚ 0.0546875 disk.rbd1.in‚0 disk.sda.in‚ 0.06640625 disk.dm-1.out‚ 8070.40625$ memory_percent‚96.83310985565186* disk./.usagePercent‚15.938647118559423 disk.nvme0c0n1.in‚ 416.734375 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2694.4921875 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.rbd1.out‚ 22.197265625 disk.nvme0n1.in‚ 416.734375 disk.drbd2080.out‚0# disk.dm-0.out‚40968.56494140625) proc.memory.percent‚16.44587516784668 network.sent‚ 2.6566397e+07 network.recv‚ 2.9915e+06 disk.rbd0.out‚ 1.0986328125 disk.dm-0.in‚ 416.734375 disk.sda3.in‚ 0.06640625& disk.nvme0n1.out‚40968.56494140625 disk.drbd2080.in‚0 disk.sda.out‚ 8070.40625 disk.sda2.in‚0 disk.sda1.in‚0 disk.sda1.out‚0$ disk./.usageGB‚237.4973602294922 disk.sda3.out‚ 8070.40625( disk.nvme0c0n1.out‚40968.56494140625 disk.drbd2266.in‚0( proc.memory.availableMB‚ 518.86328125 proc.cpu.threads‚17ùo­£'‚(:’ ¿ÀãÔ•¤¢® gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚136.444' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.288799999999995 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Þ)à‡¤'‚(:ü ¿ÀãÔ¾¦Œ° disk.rbd0.in‚ 0.0546875) proc.memory.percent‚16.44587516784668 network.sent‚ 2.6572656e+07 network.recv‚ 2.996154e+06$ disk./.usageGB‚237.4973602294922 disk.drbd2266.out‚0 disk.drbd2265.out‚0' proc.memory.availableMB‚ 518.6171875 disk.drbd2266.in‚0 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 416.734375 cpu‚25.350212094006878 disk.dm-1.out‚ 8070.796875 disk.rbd0.out‚ 1.0986328125 proc.cpu.threads‚17 disk.dm-1.in‚ 0.06640625 disk.drbd2080.out‚0 disk.sda.in‚ 0.06640625" proc.memory.rssMB‚ 2694.4921875* disk./.usagePercent‚15.938647118559423" disk.dm-0.out‚40969.0166015625 disk.sda1.out‚0 disk.rbd1.out‚ 22.345703125% disk.nvme0n1.out‚40969.0166015625 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.sda3.out‚ 8070.796875 disk.rbd1.in‚0 disk.sda.out‚ 8070.796875# memory_percent‚96.8346118927002 disk.drbd1716.in‚0 disk.dm-0.in‚ 416.734375 disk.sda3.in‚ 0.06640625' disk.nvme0c0n1.out‚40969.0166015625 disk.nvme0n1.in‚ 416.734375 disk.drbd1716.out‚0 disk.drbd2265.in‚0–ͦ2¥'- _wandbruntime‚8402  _runtime‚8402iÆî"Ȧ' _step‚880 & lr-AdamW/pg1‚7.956810724341259e-06 & lr-AdamW/pg2‚7.956810724341259e-06 trainer/global_step‚10749 % _timestamp‚1.7884037778853765e+09 _runtime‚8402.522254105Ç­7ìͧ'Ç & lr-AdamW/pg1‚7.956810724341259e-06 & lr-AdamW/pg2‚7.956810724341259e-06 trainer/global_step‚10749 % _timestamp‚1.7884037778853765e+09 _runtime‚8402.522254105 _step‚880ðóØ2¨'- _wandbruntime‚8402  _runtime‚8402g$î'©'èO ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 200098816 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚303032672154676 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 143451185 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 217167279 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚302666369755136 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚302666369755136 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 217187317 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15154869002240 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15154700002304 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚317821256760320 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 217187317 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15418292222976 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15111202031668 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302666387758080 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚320705603493888 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _runtime‚8402.523323532 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82586954 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚317821069757440 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10749 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143450656 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15418492321792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚317821256760320 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 217187317 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15418292222976 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15111202031668 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚302666387758080 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚320705603493888 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _runtime‚8402.523323532 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚82586954 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚317821069757440 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚10749 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 143450656 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15418492321792 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0ñ°x€2«'- _wandbruntime‚8403  _runtime‚8403Ý-g½Á¬'òM SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚317849732260352 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15156262153728 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 141504524 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚336154434551296 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚302693639119872 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143463409 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 % _timestamp‚1.7884037785523927e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚302693639119872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141504522 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚10749 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82594331 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚303060004501764 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚302693657122816 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚287947396265984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60869600 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚320734516442112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 217187066 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 358691588 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚317849919276544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15419918109184 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15112591196420 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15156093140480 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15419715926528 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 217207130 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226122752 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚303059822340792 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 217187066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 _runtime‚8403.189793289 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 358691588 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141504522 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15156262153728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚302693657122816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143463933 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚287947413305344 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358711654 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15112426074808 _step‚882 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚320734492502016 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 358691588 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 141504522 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15156262153728 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚302693657122816 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 143463933 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚287947413305344 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 358711654 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15112426074808 _step‚882 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚320734492502016 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚336154434551296 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚302693639119872 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 143463409 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 % _timestamp‚1.7884037785523927e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚302693639119872 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 141504522 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚10749 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82594331 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚303060004501764 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚302693657122816 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚287947396265984 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚60869600 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚320734516442112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 217187066 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 358691588 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚317849919276544 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15419918109184 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15112591196420 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15156093140480 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15419715926528 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 217207130 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226122752 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚303059822340792 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 217187066 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192ò,Ó_2®'- _wandbruntime‚8403  _runtime‚8403çéñO¯'É 1loss/FBt3/assignment_loss‚0.34443512845119684 0loss/SRqqt1/detection_loss‚1.886527094244957 0loss/FRt3/assignment_loss‚2.5273497337248267 /loss/FRt4/assignment_loss‚1.231330988161704 0loss/SRqqt1/assignment_loss‚1.66651561640203 epoch‚8 0loss/SRqqt3/detection_loss‚1.823883052777361 _runtime‚8403.190995286 _step‚883 .loss/FRt2/detection_loss‚3.223957577577004 0loss/FRt1/assignment_loss‚3.6079022125764326 /loss/FRt2/assignment_loss‚3.939496228328118 2loss/SRqqt3/assignment_loss‚1.2821696974785515 0loss/FBt4/assignment_loss‚0.2112199031585976 .loss/FRt3/detection_loss‚2.537688473375832 -loss/FBt3/detection_loss‚2.46515515115526 1loss/SRqqt4/assignment_loss‚1.069865650458068 /loss/FBt1/assignment_loss‚0.310480486992414 .loss/FRt1/detection_loss‚2.062933488054709 0loss/SRqqt2/detection_loss‚2.125309305058585 0loss/SRqqt4/detection_loss‚1.070558269979892 0loss/FBt2/assignment_loss‚0.6120242506265299 .loss/FBt1/detection_loss‚2.686597670827593 /loss/FBt2/detection_loss‚3.9759623289108275 2loss/SRqqt2/assignment_loss‚2.0617427757630744 /loss/FRt4/detection_loss‚1.4418818755075336 $loss/total_loss‚44.8862391398356 $ _timestamp‚1.788403778554228e+09 trainer/global_step‚10749 /loss/FBt4/detection_loss‚0.7212521802445037'"·«T°'Î trainer/global_step‚10749 /loss/FBt4/detection_loss‚0.7212521802445037 1loss/FBt3/assignment_loss‚0.34443512845119684 0loss/SRqqt1/detection_loss‚1.886527094244957 0loss/FRt3/assignment_loss‚2.5273497337248267 /loss/FRt4/assignment_loss‚1.231330988161704 0loss/SRqqt1/assignment_loss‚1.66651561640203 epoch‚8 0loss/SRqqt3/detection_loss‚1.823883052777361 _runtime‚8403.190995286 _step‚883 .loss/FRt2/detection_loss‚3.223957577577004 0loss/FRt1/assignment_loss‚3.6079022125764326 /loss/FRt2/assignment_loss‚3.939496228328118 2loss/SRqqt3/assignment_loss‚1.2821696974785515 0loss/FBt4/assignment_loss‚0.2112199031585976 .loss/FRt3/detection_loss‚2.537688473375832 -loss/FBt3/detection_loss‚2.46515515115526 1loss/SRqqt4/assignment_loss‚1.069865650458068 /loss/FBt1/assignment_loss‚0.310480486992414 .loss/FRt1/detection_loss‚2.062933488054709 0loss/SRqqt2/detection_loss‚2.125309305058585 0loss/SRqqt4/detection_loss‚1.070558269979892 0loss/FBt2/assignment_loss‚0.6120242506265299 .loss/FBt1/detection_loss‚2.686597670827593 /loss/FBt2/detection_loss‚3.9759623289108275 2loss/SRqqt2/assignment_loss‚2.0617427757630744 /loss/FRt4/detection_loss‚1.4418818755075336 $loss/total_loss‚44.8862391398356 $ _timestamp‚1.788403778554228e+09óõK!.’±'‚(:‡ ÎÀãԶܟ® gpu.0.gpu‚27.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚124.046' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.8092 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0…ç熲'‚(:…  ÎÀãÔíѲ°( disk.nvme0c0n1.out‚40969.45849609375& disk.nvme0n1.out‚40969.45849609375$ memory_percent‚96.83339595794678 network.recv‚ 2.999872e+06 disk.rbd1.out‚ 22.345703125 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2694.4921875 disk.sda2.in‚0 disk.nvme0c0n1.in‚ 416.734375 disk.rbd0.in‚ 0.0546875# disk.dm-0.out‚40969.45849609375( proc.memory.availableMB‚ 518.81640625) proc.memory.percent‚16.44587516784668 proc.cpu.threads‚17 disk.sda1.out‚0 disk.sda3.out‚ 8071.2734375 network.sent‚ 2.6578056e+07 disk.sda.out‚ 8071.2734375 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 cpu‚25.349676661841634 disk.dm-1.out‚ 8071.2734375 disk.rbd0.out‚ 1.0986328125 disk.sda3.in‚ 0.06640625* disk./.usagePercent‚15.938650446657368% disk./.usageGB‚237.49740982055664 disk.sda.in‚ 0.06640625 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.nvme0n1.in‚ 416.734375 disk.drbd2080.out‚0 disk.dm-0.in‚ 416.734375 disk.sda1.in‚0 disk.sda2.out‚0?äZC³'‚(:’ ÝÀãÔŽ†¥® gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚142.402' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.480399999999996 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0­!ÀF“´'‚(:ˆ  ÝÀãÔÏžè° disk.nvme0c0n1.in‚ 416.734375 disk.dm-1.out‚ 8071.87890625# disk.dm-0.out‚40970.21728515625 disk.sda3.out‚ 8071.87890625 network.recv‚ 3.005818e+06& disk.nvme0n1.out‚40970.21728515625 disk.dm-0.in‚ 416.734375 disk.rbd1.in‚0 disk.rbd1.out‚ 22.501953125) proc.memory.percent‚16.44587516784668 disk.drbd2080.out‚0 disk.sda3.in‚ 0.06640625" proc.memory.rssMB‚ 2694.4921875 disk.sda.out‚ 8071.87890625 disk.dm-1.in‚ 0.06640625 disk.drbd2266.in‚0 disk.rbd0.out‚ 1.0986328125$ memory_percent‚96.83492183685303 disk.sda1.in‚0 disk.sda1.out‚0* disk./.usagePercent‚15.938604109293673% disk./.usageGB‚237.49671936035156 disk.sda2.in‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 proc.cpu.threads‚17( disk.nvme0c0n1.out‚40970.21728515625 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.rbd0.in‚ 0.0546875 network.sent‚ 2.6686571e+07 disk.nvme0n1.in‚ 416.734375 disk.drbd2266.out‚0( proc.memory.availableMB‚ 518.56640625 cpu‚25.349083241459482 disk.sda.in‚ 0.06640625 disk.drbd1716.in‚0 disk.drbd1716.out‚0[.¡52µ'- _wandbruntime‚8437  _runtime‚8437ð|”Ðʶ'Ä ' lr-AdamW/pg1‚7.3812844680872325e-06 ' lr-AdamW/pg2‚7.3812844680872325e-06 trainer/global_step‚10799 % _timestamp‚1.7884038133255026e+09 _runtime‚8437.962351954 _step‚884ÊNÃíÏ·'É _step‚884 ' lr-AdamW/pg1‚7.3812844680872325e-06 ' lr-AdamW/pg2‚7.3812844680872325e-06 trainer/global_step‚10799 % _timestamp‚1.7884038133255026e+09 _runtime‚8437.962351954ôÙÚíK2¸'- _wandbruntime‚8437  _runtime‚8437fdì'¹'æO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15489469180928 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218159981 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚304399303089884 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 218180019 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202195968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15489266984960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚337640720086016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15224540324864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144091383 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚319254209316864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚319254209316864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚337640493949952 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15224540324864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚304399120929120 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚10799 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142124813 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61135577 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 # _timestamp‚1.78840381332618e+09 _runtime‚8437.963239709 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚304029837991936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15224371324928 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218159981 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚319254396319744 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 218180019 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142124815 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚289218612200448 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15180508728672 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15224371324928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61135046 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142124813 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460²¿Sº'ëO GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15180673850076 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚533 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15489469180928 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 218159981 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚304399303089884 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 218180019 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 202195968 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15489266984960 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚531 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚337640720086016 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15224540324864 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144091383 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚319254209316864 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚319254209316864 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚337640493949952 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15224540324864 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚304399120929120 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 trainer/global_step‚10799 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 142124813 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61135577 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 # _timestamp‚1.78840381332618e+09 _runtime‚8437.963239709 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚304029837991936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15224371324928 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 218159981 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚319254396319744 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 218180019 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142124815 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚289218612200448 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15180508728672 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15224371324928 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61135046 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 142124813 JDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 _step‚886 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 218179768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82963183 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142137474 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚319282871819776 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 360337306 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15490893768704 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15225764463104 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚289244555382784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 360317240 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚304057107356672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚319283058835968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82963181 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚322180139913216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚289244572422144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚304426453276000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142137474 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚337670827304960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 360337306 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚304057125359616 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚304057125359616 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 218199832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61140289 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚322180163853312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142137472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 % _timestamp‚1.7884038140011532e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚304057107356672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15181897893216 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 218179768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚337671057622016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144103470 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61140784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚495 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 144103967 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160Züÿý&½'÷M I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15225933476352 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 360317240 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚8438.638571466 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15225933476352 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 _step‚886 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 218179768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚82963183 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142137474 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚319282871819776 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 360337306 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15490893768704 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15225764463104 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚289244555382784 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 360317240 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚304057107356672 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚319283058835968 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚82963181 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚322180139913216 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚289244572422144 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚304426453276000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142137474 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚337670827304960 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 360337306 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚304057125359616 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚304057125359616 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 218199832 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61140289 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚322180163853312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142137472 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 % _timestamp‚1.7884038140011532e+09 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚304057107356672 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15181897893216 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 218179768 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚337671057622016 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144103470 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61140784 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚495 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 144103967 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 218199832 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚304426635436972 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 FDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83324656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15250145668484 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚339157113197056 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320687348876288 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15294211647488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚339157343527424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142757765 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 219152683 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15249980547080 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _runtime‚ 8473.43136797 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚305393306228736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 219172721 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 361930486 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144733061 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚305393324231680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚290515788356608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61408405 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15560238820864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚305393324231680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61407845 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15560445211136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚290515771317248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚562 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 % _timestamp‚1.7884038487942615e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚320687535879168 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 219152683 §DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚305393324231680 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚290515788356608 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚61408405 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15560238820864 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚305393324231680 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚61407845 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15560445211136 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚290515771317248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚562 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 % _timestamp‚1.7884038487942615e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚320687535879168 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 219152683 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15294211647488 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 219172721 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 144732499 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚83324654 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚305765934025092 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _step‚889 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚83324656 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15250145668484 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚339157113197056 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚320687348876288 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15294211647488 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚339157343527424 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 142757765 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 219152683 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15249980547080 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _runtime‚ 8473.43136797 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚305393306228736 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 219172721 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 361930486 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 144733061 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚305420575593472 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚305420593596416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚323625811264512 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 361942892 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚305793084211208 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚305793266372180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61413366 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚320716198395392 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚547 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15295435785728 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚320716011379200 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 361962958 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚320716198395392 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142770422 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142770424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚290541731538944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15251369711624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 361962958 $ _timestamp‚1.788403849467289e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144745397 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _step‚890 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 361942892 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚339187441609728 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142770424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61413911 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15561869050880 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocate·g±T„d‚15295604798976 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 219192534 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15561654285312 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219192534 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚323625787324416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚290541714499584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚545 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 trainer/global_step‚10849 _runtime‚8474.104647283 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚339187680315392Q›¬ü&Í'öM YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15251369711624 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 361962958 $ _timestamp‚1.788403849467289e+09 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 144745397 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 _step‚890 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 361942892 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚339187441609728 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 142770424 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚61413911 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15561869050880 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15295604798976 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 219192534 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15561654285312 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 219192534 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚323625787324416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚290541714499584 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚545 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 trainer/global_step‚10849 _runtime‚8474.104647283 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚339187680315392 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15251534833236 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚83332031 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15295435785728 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚305420575593472 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚305420593596416 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚323625811264512 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 361942892 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚305793084211208 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚305793266372180 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚61413366 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚320716198395392 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚547 EDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15295435785728 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚320716011379200 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 361962958 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚320716198395392 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 142770422 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 142770424 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚290541731538944úð7ã2Î'- _wandbruntime‚8474  _runtime‚8474_‰OdVÏ'Ð 0loss/FBt3/assignment_loss‚0.8845044448971748 1loss/SRqqt3/detection_loss‚2.0239226069787275 epoch‚8 1loss/SRqqt4/detection_loss‚0.9483840674511157 0loss/FBt4/detection_loss‚0.49255210893271967 &loss/total_loss‚43.527604419600216 _step‚891 % _timestamp‚1.7884038494688625e+09 -loss/FBt3/detection_loss‚1.96503037876553 /loss/FRt1/detection_loss‚2.3369499935609537 0loss/SRqqt1/detection_loss‚1.907341684613909 0loss/SRqqt2/detection_loss‚2.320279425010085 0loss/FRt4/assignment_loss‚1.6015374585986137 0loss/FBt2/assignment_loss‚0.5306844590231776 /loss/FRt1/assignment_loss‚2.894224464893341 2loss/SRqqt1/assignment_loss‚1.5672463555725253 1loss/SRqqt2/assignment_loss‚2.042561132228002 2loss/SRqqt3/assignment_loss‚1.3633089624345303 0loss/FBt1/assignment_loss‚0.2517599587391301 /loss/FRt4/detection_loss‚1.9865211819298565 /loss/FBt2/detection_loss‚2.4971787482500076 _runtime‚8474.105579311 /loss/FRt3/assignment_loss‚3.326840975705315 1loss/SRqqt4/assignment_loss‚1.217942080955254 1loss/FBt4/assignment_loss‚0.16991422177046867 .loss/FRt2/detection_loss‚2.953942507505417 -loss/FRt3/detection_loss‚2.77390921291183 .loss/FBt1/detection_loss‚2.039346694946289 trainer/global_step‚10849 /loss/FRt2/assignment_loss‚3.431721293926239š7”[Ð'Õ /loss/FRt2/assignment_loss‚3.431721293926239 0loss/FBt3/assignment_loss‚0.8845044448971748 1loss/SRqqt3/detection_loss‚2.0239226069787275 epoch‚8 1loss/SRqqt4/detection_loss‚0.9483840674511157 0loss/FBt4/detection_loss‚0.49255210893271967 &loss/total_loss‚43.527604419600216 _step‚891 % _timestamp‚1.7884038494688625e+09 -loss/FBt3/detection_loss‚1.96503037876553 /loss/FRt1/detection_loss‚2.3369499935609537 0loss/SRqqt1/detection_loss‚1.907341684613909 0loss/SRqqt2/detection_loss‚2.320279425010085 0loss/FRt4/assignment_loss‚1.6015374585986137 0loss/FBt2/assignment_loss‚0.5306844590231776 /loss/FRt1/assignment_loss‚2.894224464893341 2loss/SRqqt1/assignment_loss‚1.5672463555725253 1loss/SRqqt2/assignment_loss‚2.042561132228002 2loss/SRqqt3/assignment_loss‚1.3633089624345303 0loss/FBt1/assignment_loss‚0.2517599587391301 /loss/FRt4/detection_loss‚1.9865211819298565 /loss/FBt2/detection_loss‚2.4971787482500076 _runtime‚8474.105579311 /loss/FRt3/assignment_loss‚3.326840975705315 1loss/SRqqt4/assignment_loss‚1.217942080955254 1loss/FBt4/assignment_loss‚0.16991422177046867 .loss/FRt2/detection_loss‚2.953942507505417 -loss/FRt3/detection_loss‚2.77390921291183 .loss/FBt1/detection_loss‚2.039346694946289 trainer/global_step‚10849û²ñŒ’Ñ'‚(:‡ ŠÁãÔÔ°ò­ gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚122.684' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.5368 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0mV þ”Ò'‚(:‰  ŠÁãÔÄÐÿ° disk.rbd1.in‚0 disk.sda2.in‚0 disk.dm-1.out‚ 8073.60546875 disk.drbd2080.out‚0 disk.rbd0.out‚ 1.0986328125 disk.sda3.out‚ 8073.60546875 disk.drbd1716.out‚0) proc.memory.percent‚16.44587516784668 disk.dm-1.in‚ 0.06640625 disk.drbd2080.in‚0 disk.rbd1.out‚ 22.5341796875( disk.nvme0c0n1.out‚40971.89599609375 disk.sda.out‚ 8073.60546875 disk.nvme0n1.in‚ 416.734375 disk.sda2.out‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17% disk./.usageGB‚237.49667739868164 disk.sda3.in‚ 0.06640625" proc.memory.rssMB‚ 2694.4921875 cpu‚25.347469343702272 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.dm-0.in‚ 416.734375 disk.sda.in‚ 0.06640625 disk.drbd1716.in‚0$ memory_percent‚96.83454036712646( proc.memory.availableMB‚ 518.62890625 network.sent‚ 2.6807152e+07 network.recv‚ 3.020166e+06 disk.rbd0.in‚ 0.0546875# disk.dm-0.out‚40971.89599609375 disk.sda1.in‚0 disk.sda1.out‚0 disk.nvme0c0n1.in‚ 416.734375& disk.nvme0n1.out‚40971.89599609375* disk./.usagePercent‚15.938601293210795•Š:’Ó'‚(:‡ ™ÁãԘᕮ gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚144.528' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.9056 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0'ìŸR‘Ô'‚(:†  ™ÁãÔú¿Ç°% disk.nvme0n1.out‚40972.8818359375 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.drbd2080.in‚0" disk.dm-0.out‚40972.8818359375 disk.drbd1716.in‚0 network.sent‚ 2.6915578e+07 disk.dm-0.in‚ 416.734375 disk.nvme0c0n1.in‚ 416.734375 disk.sda2.out‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2265.out‚0 cpu‚25.346948000880094 network.recv‚ 3.026258e+06 disk.sda1.out‚0 disk.sda.in‚ 0.06640625 disk.drbd2266.in‚0 disk.drbd2080.out‚0" proc.memory.rssMB‚ 2694.4921875* disk./.usagePercent‚15.938602573248467 disk.sda3.in‚ 0.06640625$ memory_percent‚96.83334827423096( proc.memory.availableMB‚ 518.82421875) proc.memory.percent‚16.44587516784668 proc.cpu.threads‚17% disk./.usageGB‚237.49669647216797 disk.sda1.in‚0 disk.sda3.out‚ 8074.10546875 disk.nvme0n1.in‚ 416.734375 disk.rbd0.out‚ 1.0986328125 disk.sda.out‚ 8074.10546875 disk.rbd1.out‚ 22.6904296875 disk.sda2.in‚0 disk.dm-1.out‚ 8074.10546875 disk.drbd2265.in‚0 disk.rbd0.in‚ 0.0546875 disk.rbd1.in‚0' disk.nvme0c0n1.out‚40972.8818359375ƒÆ‘Õ'‚(:† ¨ÁãÔ…ê¼® gpu.0.gpu‚41.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚122.395' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.479 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ŽÁŸ¬Ö'‚(:…  ¨ÁãÔšÿá° disk.sda1.out‚0 disk.nvme0c0n1.in‚ 416.734375( disk.nvme0c0n1.out‚40973.28466796875 disk.dm-1.out‚ 8074.86328125$ memory_percent‚96.83551788330078% proc.memory.availableMB‚ 518.46875 disk.sda1.in‚0& disk.nvme0n1.out‚40973.28466796875 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2694.4921875% disk./.usageGB‚237.49679946899414 disk.drbd2080.out‚0 disk.sda3.in‚ 0.06640625 disk.drbd2265.in‚0) proc.memory.percent‚16.44587516784668 disk.drbd2080.in‚0 disk.rbd1.out‚ 22.6904296875 disk.sda2.out‚0 cpu‚25.351905238152405 proc.cpu.threads‚17 network.recv‚ 3.030549e+06 network.sent‚ 2.6921601e+07 disk.rbd0.out‚ 1.0986328125# disk.dm-0.out‚40973.28466796875 disk.sda3.out‚ 8074.86328125 disk.nvme0n1.in‚ 416.734375 disk.sda.out‚ 8074.86328125 disk.sda.in‚ 0.06640625 disk.sda2.in‚0 disk.rbd0.in‚ 0.0546875 disk.dm-0.in‚ 416.734375 disk.rbd1.in‚0 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2266.in‚0 disk.drbd2265.out‚0) disk./.usagePercent‚15.93860948545189+\´Ò’×'‚(:‡ ·ÁãÔ¡ñ›® gpu.0.gpu‚41.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚122.154' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.4308 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0²ŠòŠØ'‚(:ÿ ·ÁãÔÉ ± disk.dm-1.in‚ 0.06640625) disk./.usagePercent‚15.93860948545189 disk.drbd2080.in‚0" disk.dm-0.out‚40973.9228515625 disk.rbd1.out‚ 22.72265625% disk.nvme0n1.out‚40973.9228515625 network.sent‚ 2.692786e+07 disk.drbd2080.out‚0 disk.nvme0c0n1.in‚ 416.734375 disk.sda2.out‚0" proc.memory.rssMB‚ 2694.4921875 disk.rbd0.in‚ 0.0546875 disk.sda1.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 8075.23828125 disk.rbd1.in‚0' disk.nvme0c0n1.out‚40973.9228515625 disk.nvme0n1.in‚ 416.734375 disk.drbd2266.in‚0 disk.drbd2266.out‚0 proc.cpu.threads‚17 disk.drbd1716.out‚0 disk.dm-1.out‚ 8075.23828125 disk.sda3.in‚ 0.06640625 disk.rbd0.out‚ 1.0986328125 disk.dm-0.in‚ 416.734375$ memory_percent‚96.83551788330078% proc.memory.availableMB‚ 518.46875) proc.memory.percent‚16.44587516784668 disk.sda.in‚ 0.06640625 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 network.recv‚ 3.035058e+06 disk.sda.out‚ 8075.23828125 cpu‚25.360471276566646% disk./.usageGB‚237.49679946899414 disk.sda2.in‚0 @-p’Ù'‚(:‡ ÆÁãÔ§¸–® gpu.0.gpu‚15.0 gpu.0.memory‚0, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚130.686' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.1372 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0€u±nˆÚ'‚(:ý ÆÁãÔò½õ° disk.sda.in‚ 0.06640625 disk.sda.out‚ 8075.65625 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 518.70703125 cpu‚25.368165204329948 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2265.out‚0 network.recv‚ 3.038562e+06% disk./.usageGB‚237.49680709838867 disk.dm-1.out‚ 8075.65625# disk.dm-0.out‚40974.43115234375 disk.nvme0c0n1.in‚ 416.734375( disk.nvme0c0n1.out‚40974.43115234375& disk.nvme0n1.out‚40974.43115234375 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚ 0.0546875 disk.sda1.out‚0 disk.nvme0n1.in‚ 416.734375$ memory_percent‚96.83406352996826 disk.sda2.in‚0 disk.dm-0.in‚ 416.734375 disk.sda3.out‚ 8075.65625 disk.rbd1.in‚0 network.sent‚ 2.6935512e+07 disk.drbd2266.out‚0 disk.rbd0.out‚ 1.0986328125 disk.rbd1.out‚ 22.72265625) proc.memory.percent‚16.44587516784668 proc.cpu.threads‚17 disk.drbd2265.in‚0" proc.memory.rssMB‚ 2694.4921875 disk.sda2.out‚0 disk.sda3.in‚ 0.06640625) disk./.usagePercent‚15.938609997466969júâ2Û'- _wandbruntime‚8544  _runtime‚8544·€²[T*Ü'ÏT [LDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZLDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.allocated‚ 143687741 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 220153073 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.freed‚83879218 OADeviceStatsMonitor.on_validation_batch_start/allocation.all.freed‚ 363840812 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 107572736 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚61783579 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2460 QGDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.peak‚24128 D>DeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 4390226944 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚416 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 5041553408 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚307190338007040 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚15355920676352 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚292223146320896 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 % _timestamp‚1.7884039198572423e+09 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 143687741 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 123925592 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚199 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 410712064 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚325482687372288 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 4757245257 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚217 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 4912020992 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 5041553408 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 220169050 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 145311232 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 127308288 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚322546258683392 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚15355920676352 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 145662516 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚341105633358336 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 410712064 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚15623077498880 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 [LDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.peak‚ 4390226944 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚416 XIDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.allocated‚ 5041553408 RLDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.freed‚0 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚307190338007040 aNDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.allocated‚15355920676352 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚292223146320896 D>DeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 % _timestamp‚1.7884039198572423e+09 VHDeviceStatsMonitor.on_validation_batch_start/active.large_pool.allocated‚ 143687741 MGDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.freed‚0 ]ODeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.current‚ 123925592 OGDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.current‚199 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 410712064 jVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.allocated‚325482687372288 ;5DeviceStatsMonitor.on_validation_batch_start/num_ooms‚0 TEDeviceStatsMonitor.on_validation_batch_start/requested_bytes.all.peak‚ 4757245257 LFDeviceStatsMonitor.on_validation_batch_start/oversize_segments.current‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.large_pool.peak‚217 TEDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.peak‚ 4912020992 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 VGDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.all.current‚ 5041553408 VHDeviceStatsMonitor.on_validation_batch_start/active.small_pool.allocated‚ 220169050 SEDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.current‚ 145311232 YLDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.current‚ 127308288 ^JDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.allocated‚322546258683392 dQDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.allocated‚15355920676352 SEDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.freed‚ 145662516 _KDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.freed‚341105633358336 ZLDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.peak‚ 410712064 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚15623077498880 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚985 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚282 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 269877760 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 220176446 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚15356142503936 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚307209203192832 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚15356269812224 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 145673444 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚322565473005056 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 127308288 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚292241047874560 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚416 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 145311232 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 363874899 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 145673160 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚83885718 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 220176446 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 4757245257 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚292241030835200 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 5041553408 BDeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚985 F>DeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 TLDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.current‚282 VHDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.peak‚ 269877760 TFDeviceStatsMonitor.on_validation_batch_end/active.small_pool.allocated‚ 220176446 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚15356142503936 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.freed‚0 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚307209203192832 bODeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.allocated‚15356269812224 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 145673444 ]ODeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 YEDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.allocated‚322565473005056 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 127308288 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚292241047874560 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚416 TFDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.current‚ 145311232 M?DeviceStatsMonitor.on_validation_batch_end/active.all.allocated‚ 363874899 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 145673160 WJDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.freed‚83885718 @:DeviceStatsMonitor.on_validation_batch_end/num_device_free‚0 XJDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.allocated‚ 220176446 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 4757245257 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚292241030835200 VGDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.allocated‚ 5041553408 B&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_3_of_7‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5583711266517639 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1495978832244873 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1471446305513382 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8624702095985413 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.44284576177597046 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08510638028383255 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4058063328266144 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9966107606887817 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.09375 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4842502474784851 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.527385950088501 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02083333395421505 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01950467750430107 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.550381064414978 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4955641031265259 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 trainer/global_step‚10880 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3632715940475464 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.24988920986652374 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.989201009273529 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5502212047576904 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4585636854171753 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4883929491043091 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25833332538604736 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7509255409240723 ,particle/sensitivity‚0.30170300602912903 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4566666781902313 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 epoch‚8 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5855910181999207 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 _runtime‚8558.105512914 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18214933574199677 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009448639117181301 *jet/accuracy_1_of_4‚0.9124464988708496 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6975228190422058 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_4‚1 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5939668416976929 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9767857193946838 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.021859604865312576 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09999076277017593 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1461123675107956 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3263888955116272 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.40435734391212463 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.260486364364624 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 particle/accuracy_5_of_7‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3779069781303406 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5723720192909241 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.37105655670166016 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27161088585853577 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9933592677116394 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8178697228431702 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15482491254806519 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.19212962687015533 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9851409196853638 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07173019647598267 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3788287937641144 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6491227746009827 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6005036234855652 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12878787517547607 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4796587824821472 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4682817757129669 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,jet/accuracy_4_of_4‚0.006044226698577404 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_7‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.21338385343551636 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9887130260467529 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6038194298744202 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5178571343421936 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 *jet/accuracy_2_of_2‚0.1158633679151535 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 particle/accuracy_1_of_7‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4195042848587036 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_6‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.26498597860336304 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04583333060145378 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.40477821230888367 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5079736113548279 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.30955958366394043 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4134099781513214 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.43646520376205444 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20697104930877686 particle/accuracy_2_of_5‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4992825388908386 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3772321343421936 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,jet/accuracy_4_of_5‚0.004480286967009306 particle/accuracy_2_of_4‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4670262038707733 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.48643219470977783 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5566391944885254 jet/accuracy_1_of_6‚1 jet/accuracy_5_of_7‚0 particle/accuracy_6_of_7‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17639803886413574 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_5‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4285714328289032 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4845527708530426 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 8"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.540123462677002 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5833333134651184 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11645447462797165 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14320683479309082 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18964889645576477 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5625 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5026130080223083 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5839029550552368 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.6001471877098083 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.984872043132782 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04166936129331589 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1480553150177002 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17639803886413574 5 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.27182537317276 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.10823755711317062 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20430687069892883 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05921052768826485 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 particle/accuracy_4_of_6‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.05094339698553085 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚44.04174048926668 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9774546027183533 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0028083028737455606 +jet/accuracy_3_of_6‚0.18589742481708527 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 *jet/accuracy_3_of_4‚0.0742606371641159 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9927052855491638 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013653147034347057 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.41228073835372925 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5529761910438538 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9467432498931885 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4846729040145874 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 jet/accuracy_4_of_6‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4157288074493408 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.18189655244350433 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3684028089046478 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4057454466819763 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3581833243370056 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07979752123355865 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.2954545319080353 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25214219093322754 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15759748220443726 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22192791104316711 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_4‚0.37977057695388794 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22880791127681732 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 particle/accuracy_4_of_5‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1388888955116272 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9935721158981323 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1291554570198059 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9850499629974365 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_2_of_5‚0.4487262964248657 +jet/accuracy_3_of_5‚0.11003583669662476 particle/accuracy_2_of_6‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5271542072296143 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 8"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.538690447807312 +jet/accuracy_2_of_3‚0.23931868374347687 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4399999976158142 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267138838768005 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5665479302406311 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7424241900444031 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722350239753723 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03201504424214363 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6187388896942139 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4221491515636444 particle/accuracy_1_of_2‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5040755867958069 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.545976996421814 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.48787015676498413 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5845456123352051 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5071145296096802 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19429336488246918 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5207515358924866 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16231462359428406 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.506233274936676 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.534514844417572 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.37199151515960693 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6523233652114868 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20000210404396057 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.22860360145568848 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5141814351081848 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9907435774803162 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1652824580669403 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4084967374801636 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03488372266292572 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.40619343519210815 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9913015961647034 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06578947603702545 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.326921284198761 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9773584604263306 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4099598824977875 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12372449040412903 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1088888943195343 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.24907413125038147 *jet/accuracy_2_of_6‚0.4871794581413269 jet/accuracy_6_of_7‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_7‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.961403489112854 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5124582052230835 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5833333134651184 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 jet/accuracy_1_of_7‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1660100519657135 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_2‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5536667108535767 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35728126764297485 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_3_of_4‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17154112458229065 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998068630695343 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_1_of_5‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16412439942359924 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1836034655570984 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.022152934223413467 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5046296715736389 particle/accuracy_4_of_7‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3001473844051361 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9959068298339844 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 -"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_7‚0 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.389552503824234 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1071428582072258 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.32499998807907104 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 % _timestamp‚1.7884039334663575e+09 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9905487298965454 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/2FRt2SRq¸{*üêqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912897944450378 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_7‚1 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.020653817802667618 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07887837290763855 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35263973474502563 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5249999761581421 8"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.376767635345459 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9972660541534424 (particle/accuracy‚0.8460027575492859 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15232305228710175 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 _step‚894 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10718189179897308 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.148878276348114 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9875830411911011 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 (particle/f_score‚0.37938591837882996 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.029428690671920776 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1388888955116272 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.258438378572464 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16099973022937775 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 *jet/accuracy_1_of_5‚0.9368152022361755 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814236760139465 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5514770150184631 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2183907926082611 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 particle/accuracy_7_of_7‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1900622844696045 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.017786739394068718 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.49573424458503723 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.19619882106781006 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5375352501869202 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21800240874290466 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5769520998001099 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.007999495603144169 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18599534034729004 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.971875011920929 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5168308019638062 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3194444477558136 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5040755867958069 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4726017117500305 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1942010372877121 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3958333432674408 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1U_iå'‚¦ particle/accuracy_6_of_7‚1 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17639803886413574 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 particle/accuracy_2_of_3‚1 particle/accuracy_3_of_5‚1 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.4285714328289032 9"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.4845527708530426 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 8"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.540123462677002 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 jet/accuracy_5_of_5‚0 9"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.5833333134651184 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11645447462797165 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.14320683479309082 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18964889645576477 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5625 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5026130080223083 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5839029550552368 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.6001471877098083 6 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.984872043132782 :"Purity/1FRt1SRqqt1FBt/event_purity‚0.04166936129331589 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1480553150177002 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17639803886413574 5 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.27182537317276 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.10823755711317062 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 particle/accuracy_3_of_3‚1 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 :"Purity/1FRt0SRqqt*FBt/event_purity‚0.20430687069892883 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05921052768826485 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2916666567325592 particle/accuracy_4_of_6‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 :"Purity/0FRt2SRqqt1FBt/event_purity‚0.05094339698553085 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 )loss/val_total_loss‚44.04174048926668 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚0 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9774546027183533 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0028083028737455606 +jet/accuracy_3_of_6‚0.18589742481708527 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9929894208908081 *jet/accuracy_3_of_4‚0.0742606371641159 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9927052855491638 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.013653147034347057 :"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.41228073835372925 9"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.5529761910438538 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 +particle/specificity‚0.9467432498931885 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4846729040145874 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 jet/accuracy_4_of_6‚0 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 9"Purity/0FRt1SRqqt*FBt/event_purity‚0.4157288074493408 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.18189655244350433 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 7 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.3684028089046478 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 *"Purity/4FRt*SRqqt*FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4057454466819763 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt0SRqqt2FBt/event_purity‚0.3581833243370056 :"Purity/1FRt*SRqqt0FBt/event_purity‚0.07979752123355865 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.2954545319080353 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 :"Purity/*FRt*SRqqt*FBt/event_purity‚0.25214219093322754 8 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.15759748220443726 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22192791104316711 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 ("Purity/1FRt2SRqqt0FBt/event_purity‚0 +jet/accuracy_2_of_4‚0.37977057695388794 ."Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.40625 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.22880791127681732 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 particle/accuracy_4_of_5‚1 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1388888955116272 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9935721158981323 7 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.1291554570198059 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 7 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.9850499629974365 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 *jet/accuracy_2_of_5‚0.4487262964248657 +jet/accuracy_3_of_5‚0.11003583669662476 particle/accuracy_2_of_6‚1 ) Purity/4FRt*SRqqt*FBt/FRt_purity‚0.25 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5271542072296143 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 8"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.538690447807312 +jet/accuracy_2_of_3‚0.23931868374347687 9"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.4399999976158142 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267138838768005 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5665479302406311 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7424241900444031 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9722350239753723 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03201504424214363 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6187388896942139 7 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.4221491515636444 particle/accuracy_1_of_2‚1 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5040755867958069 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.545976996421814 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.48787015676498413 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5845456123352051 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5071145296096802 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ( Purity/4FRt1SRqqt*FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19429336488246918 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5207515358924866 8 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.16231462359428406 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 8"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.506233274936676 8"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.534514844417572 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.37199151515960693 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6523233652114868 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 8 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.20000210404396057 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.22860360145568848 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5141814351081848 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9907435774803162 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1652824580669403 9"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.4084967374801636 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03488372266292572 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9922481179237366 particle/accuracy_1_of_3‚1 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.40619343519210815 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9913015961647034 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.06578947603702545 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 6 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.326921284198761 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.9773584604263306 7 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.4099598824977875 8 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.12372449040412903 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 7 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.1088888943195343 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.7727272510528564 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 & Purity/1FRt4SRqqt*FBt/FRt_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.24907413125038147 *jet/accuracy_2_of_6‚0.4871794581413269 jet/accuracy_6_of_7‚0 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 jet/accuracy_3_of_7‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 6 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.961403489112854 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5124582052230835 9"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.5833333134651184 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 jet/accuracy_1_of_7‚1 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1660100519657135 particle/accuracy_1_of_1‚1 particle/accuracy_2_of_2‚1 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.4444444477558136 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5536667108535767 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35728126764297485 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 particle/accuracy_3_of_4‚1 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17154112458229065 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998068630695343 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 particle/accuracy_1_of_5‚1 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.16412439942359924 7 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.1836034655570984 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.022152934223413467 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5046296715736389 particle/accuracy_4_of_7‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 9"Purity/1FRt0SRqqt1FBt/event_purity‚0.3001473844051361 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9959068298339844 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 -"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5625 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 jet/accuracy_4_of_7‚0 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.389552503824234 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt0SRqqt3FBt/event_purity‚0.1071428582072258 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.32499998807907104 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 % _timestamp‚1.7884039334663575e+09 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9905487298965454 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912897944450378 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 particle/accuracy_2_of_7‚1 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.020653817802667618 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.07887837290763855 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35263973474502563 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5249999761581421 8"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.376767635345459 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9972660541534424 (particle/accuracy‚0.8460027575492859 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_1_of_6‚1 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.15232305228710175 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 ,"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.625 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 _step‚894 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.10718189179897308 8"Purity/*FRt0SRqqt0FBt/event_purity‚0.148878276348114 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9875830411911011 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 (particle/f_score‚0.37938591837882996 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 ;"Purity/*FRt2SRqqt*FBt/event_purity‚0.029428690671920776 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1388888955116272 8"Purity/*FRt0SRqqt2FBt/event_purity‚0.258438378572464 8 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.16099973022937775 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 *jet/accuracy_1_of_5‚0.9368152022361755 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9814236760139465 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5514770150184631 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_4‚1 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.2183907926082611 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 particle/accuracy_7_of_7‚1 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1900622844696045 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.017786739394068718 8 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.49573424458503723 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.19619882106781006 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5375352501869202 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21800240874290466 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 particle/accuracy_5_of_6‚1 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5769520998001099 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.007999495603144169 :"Purity/*FRt*SRqqt0FBt/event_purity‚0.18599534034729004 6 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.971875011920929 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5168308019638062 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.3194444477558136 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.3333333432674408 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5040755867958069 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 9"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.3499999940395355 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 9"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.4726017117500305 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1942010372877121 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9886363744735718 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3958333432674408 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 ,jet/accuracy_3_of_3‚0.021389639005064964 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19660747051239014 7 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.2693610191345215 8 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.10185184329748154 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9662356376647949 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 jet/accuracy_7_of_7‚0 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9733560085296631 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.022616524249315262 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4741319715976715 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4444445073604584 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22449137270450592 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08103477209806442 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 particle/accuracy_3_of_7‚1 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5583711266517639 7 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.1495978832244873 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 9"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.3499999940395355 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 7 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.1471446305513382 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 *jet/accuracy_1_of_3‚0.8624702095985413 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 *"Purity/4FRt*SRqqt0FBt/SRqqt_purity‚0.5 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.44284576177597046 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.08510638028383255 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 9"Purity/0FRt*SRqqt0FBt/event_purity‚0.4058063328266144 7 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.9966107606887817 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 ."Purity/0FRt0SRqqt3FBt/event_purity‚0.09375 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4842502474784851 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 8"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.527385950088501 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02083333395421505 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 :"Purity/1FRt1SRqqt*FBt/event_purity‚0.01950467750430107 8"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5503810644¶ K*414978 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ( Purity/4FRt1SRqqt0FBt/FRt_purity‚0.5 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.4955641031265259 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 trainer/global_step‚10880 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 7 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.3632715940475464 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.24988920986652374 6 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.989201009273529 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5502212047576904 9"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.4585636854171753 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4883929491043091 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 particle/accuracy_5_of_5‚1 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.25833332538604736 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 *jet/accuracy_1_of_2‚0.7509255409240723 ,particle/sensitivity‚0.30170300602912903 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 9"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.4566666781902313 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 epoch‚8 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5855910181999207 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 _runtime‚8558.105512914 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18214933574199677 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚0 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009448639117181301 *jet/accuracy_1_of_4‚0.9124464988708496 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.3333333432674408 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6975228190422058 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_4‚1 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5939668416976929 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9767857193946838 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 ) Purity/4FRt*SRqqt0FBt/FRt_purity‚0.25 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.21875 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.021859604865312576 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09999076277017593 7 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.1461123675107956 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.3263888955116272 :"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.40435734391212463 6 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.260486364364624 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 particle/accuracy_5_of_7‚1 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3779069781303406 9"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.5723720192909241 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.37105655670166016 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 jet/accuracy_5_of_6‚0 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27161088585853577 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9933592677116394 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8178697228431702 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.15482491254806519 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.19212962687015533 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9851409196853638 :"Purity/0FRt2SRqqt*FBt/event_purity‚0.07173019647598267 9"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.3788287937641144 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6491227746009827 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.6005036234855652 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.12878787517547607 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 9"Purity/0FRt*SRqqt*FBt/event_purity‚0.4796587824821472 9"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.4682817757129669 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 & Purity/1FRt4SRqqt0FBt/FRt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9979166388511658 ,jet/accuracy_4_of_4‚0.006044226698577404 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 jet/accuracy_2_of_7‚0 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 ("Purity/1FRt2SRqqt*FBt/event_purity‚0 8 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.21338385343551636 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9887130260467529 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.6038194298744202 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 jet/accuracy_6_of_6‚0 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5178571343421936 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 *jet/accuracy_2_of_2‚0.1158633679151535 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 particle/accuracy_1_of_7‚1 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4195042848587036 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 particle/accuracy_6_of_6‚1 :"Purity/*FRt1SRqqt0FBt/event_purity‚0.26498597860336304 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04583333060145378 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 :"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.40477821230888367 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5079736113548279 8 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.30955958366394043 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4134099781513214 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_6‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.43646520376205444 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20697104930877686 particle/accuracy_2_of_5‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.4992825388908386 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3772321343421936 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,jet/accuracy_4_of_5‚0.004480286967009306 particle/accuracy_2_of_4‚1 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 9"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.4670262038707733 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 +jet/accuracy_1_of_1‚0.48643219470977783 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 9"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.5566391944885254 jet/accuracy_1_of_6‚1 jet/accuracy_5_of_7‚0þ3&&áæ'‚(:’ äÁãÔÄÚ® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚141.047' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.209400000000002 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0’¬ôèŒç'‚(:  äÁãÔúä± disk.drbd2266.out‚0 disk.rbd0.out‚ 1.0986328125 disk.sda1.out‚0 disk.sda3.in‚ 0.06640625 disk.sda.in‚ 0.06640625 disk.sda.out‚ 8077.0078125* disk./.usagePercent‚15.849598993894487 disk.drbd2265.in‚0 disk.rbd1.out‚ 22.87890625 disk.nvme0c0n1.in‚ 416.734375& disk.nvme0n1.out‚41099.03173828125$ memory_percent‚97.58954048156738 disk.drbd1716.out‚0 disk.nvme0n1.in‚ 416.734375 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚ 0.0546875 disk.dm-0.in‚ 416.734375# disk.dm-0.out‚41099.03173828125 disk.sda3.out‚ 8077.0078125) proc.memory.percent‚16.44587516784668 network.sent‚ 2.704658e+07 disk.dm-1.out‚ 8077.0078125 disk.rbd1.in‚0( disk.nvme0c0n1.out‚41099.03173828125" proc.memory.rssMB‚ 2694.4921875 disk.sda1.in‚0 proc.cpu.threads‚17 network.recv‚ 3.048282e+06% disk./.usageGB‚236.17047882080078 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0' proc.memory.availableMB‚ 394.9296875 cpu‚25.38037831152692 disk.dm-1.in‚ 0.06640625›Áõ2è'- _wandbruntime‚8573  _runtime‚8573Ge•Æé'À % lr-AdamW/pg1‚6.29293713274974e-06 % lr-AdamW/pg2‚6.29293713274974e-06 trainer/global_step‚10899 % _timestamp‚1.7884039488831286e+09 _runtime‚8573.519970418 _step‚8954f‰Ëê'Å _runtime‚8573.519970418 _step‚895 % lr-AdamW/pg1‚6.29293713274974e-06 % lr-AdamW/pg2‚6.29293713274974e-06 trainer/global_step‚10899 % _timestamp‚1.7884039488831286e+09ÿïêX¿2ë'- _wandbruntime‚8573  _runtime‚8573WY¢Øî'ì'èO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚307945281170432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚326277546418176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _step‚896 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚323331194694144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144065569 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚8573.520816765 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15385726520832 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146039390 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚323331194694144 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 364678839 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 220633308 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15653406671360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144065571 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15341368672034 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚323331007691264 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15385726520832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 364698879 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚341930953089536 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚307945299173376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 220613270 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚308282279147678 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323331007691264 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚341931189711360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚292940745354240ô¡u´%í'íO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚491 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚489 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 % _timestamp‚1.7884039488838055e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 144065571 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚307945299173376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144065569 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _runtime‚8573.520816765 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15385726520832 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 146039390 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚323331194694144 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 364678839 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 220633308 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15653406671360 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144065571 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15341368672034 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚323331007691264 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15385726520832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 364698879 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚341930953089536 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚307945299173376 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 220613270 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚308282279147678 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚323331007691264 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚341931189711360 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚292940745354240 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚307945281170432 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚326277546418176 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _step‚896 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚323331194694144 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 364731351 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚341961516759040 trainer/global_step‚10899 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146051935 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚292966671497216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84110381 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 220653121 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 364711285 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15342757836578 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚307972568538112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚308309429333794 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0¤6õý&ð'÷M VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚502 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 364711285 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15387119659008 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚341961286441984 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 364731351 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 % _timestamp‚1.7884039495558121e+09 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚500 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚308309429333794 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 BDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15387288672256 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84110383 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15342922958190 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚323359670194176 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 144078230 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚323359857210368 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 220653121 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144078228 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 364731351 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚341961516759040 trainer/global_step‚10899 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146051935 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚292966671497216 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84110381 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 220653121 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 364711285 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15342757836578 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚307972568538112 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚324764147250688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚324764147250688 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 _step‚900 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84471854 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144698521 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚309308767410176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84471856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62205160 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚309648727922122 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15455397843456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚294237904471040 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15410840490442 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15455566843392 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \IDeviceStatsMonitor.on_train_batc<6 öh_start/active_bytes.small_pool.allocated‚15455566843392 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15411005611846 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 366304491 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15455397843456 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144698519 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 trainer/global_step‚10949 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚343447641178624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 146677016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚523 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 221626010 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 221626010 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚327723193829376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _runtime‚8609.326312459 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚324764334253568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚309308767410176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0S=î®ò'ý'ìO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚324764147250688 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚324764147250688 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 _step‚900 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84471854 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 144698521 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚309308767410176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84471856 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62205160 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚309648727922122 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15455397843456 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚294237904471040 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15410840490442 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15455566843392 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15455566843392 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15411005611846 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 366304491 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15455397843456 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 144698519 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 trainer/global_step‚10949 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚343447641178624 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 146677016 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚523 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 221626010 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 221626010 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚327723193829376 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _runtime‚8609.326312459 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚324764334253568 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚309308767410176 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15724423409152 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 221605972 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 366324531 JDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 221625759 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62210433 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚309676242429974 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚294263830614016 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 221645823 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚324792809753600 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚309676060269002 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15456959994880 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146689135 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚294263847653376 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144711178 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 221645823 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚343477966029824 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144711178 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚309336018771968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84479231 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚327752130717696 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚10949 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15456790981632 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚309336036774912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 366336937 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 _step‚901 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚324792996769792 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15725835312128 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 366357003 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚343477733615616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15725626838016 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217n¹3(÷M QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15456959994880 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 146689135 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚294263847653376 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 144711178 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 221645823 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚343477966029824 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 144711178 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚309336018771968 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84479231 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚327752130717696 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 trainer/global_step‚10949 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15456790981632 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚309336036774912 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 366336937 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 FDeviceStatsMonitor.on_train_batch_end/reque%eUéÊ sted_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 _step‚901 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚324792996769792 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15725835312128 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 366357003 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚343477733615616 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15725626838016 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 221625759 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62210433 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚309676242429974 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚294263830614016 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 221645823 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚324792809753600 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚309676060269002…haFà‘‚(‚(:†  ‘ÂãÔïÒܱ network.recv‚ 3.064088e+06 disk.nvme0c0n1.in‚ 416.734375( disk.nvme0c0n1.out‚41225.59228515625 disk.nvme0n1.in‚ 416.734375 disk.sda.in‚ 0.06640625 proc.cpu.threads‚17 disk.sda3.in‚ 0.06640625 disk.dm-1.in‚ 0.06640625 disk.drbd2266.in‚0$ memory_percent‚97.59132862091064 disk.rbd1.out‚ 23.0673828125 disk.drbd1716.out‚0) proc.memory.percent‚16.44587516784668& disk.nvme0n1.out‚41225.59228515625 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚ 0.0546875 disk.rbd0.out‚ 1.0986328125# disk.dm-0.out‚41225.59228515625 disk.sda3.out‚ 8078.9921875* disk./.usagePercent‚15.849369355136272% disk./.usageGB‚236.16705703735352 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 network.sent‚ 2.7284499e+07 disk.dm-1.out‚ 8078.9921875 disk.dm-0.in‚ 416.734375 disk.sda1.in‚0( proc.memory.availableMB‚ 394.63671875" proc.memory.rssMB‚ 2694.4921875 cpu‚25.378627976361297 disk.rbd1.in‚0 disk.sda.out‚ 8078.9921875 disk.sda1.out‚0á ÏÐ2ƒ(- _wandbruntime‚8610  _runtime‚8610¿f …W„(Ñ 0loss/FRt3/assignment_loss‚2.9043791874861107 .loss/FRt1/detection_loss‚2.158602415930991 /loss/FBt2/detection_loss‚2.7156136706471443 _step‚902 /loss/FRt2/assignment_loss‚3.951300806012647 1loss/SRqqt3/assignment_loss‚1.449543158555853 .loss/FRt2/detection_loss‚3.068951565643837 /loss/FBt3/detection_loss‚2.5232604609595404 0loss/FRt4/assignment_loss‚1.5171608035526578 _runtime‚8610.010020595 % _timestamp‚1.7884039853732865e+09 0loss/FRt1/assignment_loss‚3.0400077595430264 1loss/SRqqt1/assignment_loss‚1.297558270232833 -loss/FBt4/detection_loss‚0.67802433340847 %loss/total_loss‚42.21927411582736 2loss/SRqqt4/assignment_loss‚0.9258983414019307 1loss/FBt3/assignment_loss‚0.18683890429221922 1loss/FBt4/assignment_loss‚0.09736584188923221 .loss/FRt4/detection_loss‚1.687115655354564 /loss/SRqqt1/detection_loss‚1.87120220421449 1loss/SRqqt3/detection_loss‚1.5784837416019932 .loss/FBt1/detection_loss‚2.427232493956884 1loss/FBt1/assignment_loss‚0.40228157901826006 1loss/FBt2/assignment_loss‚0.28645707108009333 1loss/SRqqt4/detection_loss‚1.1126507096473248 trainer/global_step‚10949 1loss/SRqqt2/assignment_loss‚1.708882567722623 .loss/FRt3/detection_loss‚2.504931872089704 1loss/SRqqt2/detection_loss‚2.1255307015849323 epoch‚9n6¡Q\…(Ö .loss/FRt3/detection_loss‚2.504931872089704 1loss/SRqqt2/detection_loss‚2.1255307015849323 epoch‚9 0loss/FRt3/assignment_loss‚2.9043791874861107 .loss/FRt1/detection_loss‚2.158602415930991 /loss/FBt2/detection_loss‚2.7156136706471443 _step‚902 /loss/FRt2/assignment_loss‚3.951300806012647 1loss/SRqqt3/assignment_loss‚1.449543158555853 .loss/FRt2/detection_loss‚3.068951565643837 /loss/FBt3/detection_loss‚2.5232604609595404 0loss/FRt4/assignment_loss‚1.5171608035526578 _runtime‚8610.010020595 % _timestamp‚1.7884039853732865e+09 0loss/FRt1/assignment_loss‚3.0400077595430264 1loss/SRqqt1/assignment_loss‚1.297558270232833 -loss/FBt4/detection_loss‚0.67802433340847 %loss/total_loss‚42.21927411582736 2loss/SRqqt4/assignment_loss‚0.9258983414019307 1loss/FBt3/assignment_loss‚0.18683890429221922 1loss/FBt4/assignment_loss‚0.09736584188923221 .loss/FRt4/detection_loss‚1.687115655354564 /loss/SRqqt1/detection_loss‚1.87120220421449 1loss/SRqqt3/detection_loss‚1.5784837416019932 .loss/FBt1/detection_loss‚2.427232493956884 1loss/FBt1/assignment_loss‚0.40228157901826006 1loss/FBt2/assignment_loss‚0.28645707108009333 1loss/SRqqt4/detection_loss‚1.1126507096473248 trainer/global_step‚10949 1loss/SRqqt2/assignment_loss‚1.708882567722623†w¨“’†(‚(:‡  ÂãÔïÇÿ­ gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚126.172' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.2344 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 M1‹‡(‚(:€   ÂãÔÇãа disk.sda3.out‚ 8079.5859375 disk.drbd2266.in‚0 network.recv‚ 3.07027e+06$ disk./.usageGB‚236.1670684814453 disk.sda.out‚ 8079.5859375 disk.dm-1.in‚ 0.06640625 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.drbd2080.out‚0 disk.rbd0.out‚ 1.0986328125 disk.nvme0c0n1.in‚ 416.734375$ proc.memory.availableMB‚394.8125" proc.memory.rssMB‚ 2694.4921875 disk.sda2.out‚0 disk.drbd1716.in‚0 disk.dm-0.in‚ 416.734375& disk.nvme0n1.out‚41226.00341796875 disk.rbd1.in‚0* disk./.usagePercent‚15.849370123158876 disk.drbd2265.in‚0# disk.dm-0.out‚41226.00341796875 disk.nvme0n1.in‚ 416.734375 disk.sda.in‚ 0.06640625 disk.drbd2265.out‚0 disk.sda3.in‚ 0.06640625 disk.rbd1.out‚ 23.0673828125 disk.dm-1.out‚ 8079.5859375 network.sent‚ 2.7381811e+07 disk.rbd0.in‚ 0.0546875 disk.sda1.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚41226.00341796875$ memory_percent‚97.59025573730469 cpu‚25.378035850163315 disk.sda2.in‚0 disk.drbd2080.in‚0) proc.memory.percent‚16.44587516784668 proc.cpu.threads‚17õ¦\Bˆ(‚(:’ ¯ÂãÔϾš® gpu.0.gpu‚26.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚35.0 gpu.0.powerWatts‚147.487' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.497400000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0VV#~Ž‰(‚(:ƒ  ¯ÂãÔ½ºà° disk.sda1.out‚0 disk.sda3.in‚ 0.06640625 disk.sda2.in‚0" proc.memory.rssMB‚ 2694.4921875 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.rbd0.out‚ 1.0986328125 disk.sda3.out‚ 8080.12109375 disk.rbd1.out‚ 23.099609375 disk.nvme0c0n1.in‚ 416.734375) proc.memory.percent‚16.44587516784668 disk.nvme0n1.in‚ 416.734375 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.rbd1.in‚0 disk.sda.out‚ 8080.12109375 disk.dm-1.out‚ 8080.12109375' proc.memory.availableMB‚ 394.0703125 network.recv‚ 3.076224e+06 disk.rbd0.in‚ 0.0546875 disk.drbd2265.out‚0 cpu‚25.377467215618047* disk./.usagePercent‚15.849370123158876 disk.drbd2080.in‚0 disk.dm-1.in‚ 0.06640625$ disk./.usageGB‚236.1670684814453" disk.dm-0.out‚41226.6259765625 disk.sda.in‚ 0.06640625 disk.drbd1716.out‚0 disk.drbd2266.in‚0' disk.nvme0c0n1.out‚41226.6259765625% disk.nvme0n1.out‚41226.6259765625 disk.sda2.out‚0 disk.drbd1716.in‚0$ memory_percent‚97.59478569030762 network.sent‚ 2.7479774e+07 disk.dm-0.in‚ 416.734375 disk.sda1.in‚0̤G2Š(- _wandbruntime‚8644  _runtime‚8644zíÝkÉ‹(à ' lr-AdamW/pg2‚5.2889454237271765e-06 trainer/global_step‚10999 $ _timestamp‚1.788404019889254e+09 _runtime‚8644.526102924 _step‚903 ' lr-AdamW/pg1‚5.2889454237271765e-06 öbOÎŒ(È _runtime‚8644.526102924 _step‚903 ' lr-AdamW/pg1‚5.2889454237271765e-06 ' lr-AdamW/pg2‚5.2889454237271765e-06 trainer/global_step‚10999 $ _timestamp‚1.788404019889254e+09‡7ÖÎ]2(- _wandbruntime‚8644  _runtime‚8644¦„·#í'Ž(çO JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚520 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚311015358857330 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 $ _timestamp‚1.788404019889986e+09 _runtime‚8644.527013902 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚326197473812992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15525238166016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚329168865180672 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145331469 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145331471 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚310672217644032 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 222618712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚329168841240576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚326197286810112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚344963961316352 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 222618712 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 trainer/global_step‚10999 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 _step‚904 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚84840706 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 367930143 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚310672217644032 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145331469 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15480312308850 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62475680 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15480477430254 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84840704 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚326197473812992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚344964200035328 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚326197286810112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15525238166016 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15525069166080 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚295535046548480 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 222598674 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62475162 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚310672235646976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚518 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚84840704 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚326197473812992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚344964200035328 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚326197286810112 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15525238166016 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15525069166080 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚295535046548480 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 222598674 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62475162 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚310672235646976 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚518 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 $ _timestamp‚1.788404019889986e+09 _runtime‚8644.527013902 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚326197473812992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15525238166016 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚329168865180672 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145331469 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145331471 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚310672217644032 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 222618712 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚329168841240576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚326197286810112 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217ˆõ]©2(- _wandbruntime‚8645  _runtime‚8645Ä>~õ‘(òM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 367962589 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚310699505011712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚295560989730816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145344128 _runtime‚8645.191548358 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 367982655 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚326226136329216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84848083 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚310699487008768 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62480972 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145344128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_trafÐœ¢in_batch_end/requested_bytes.small_pool.freed‚15481701473394 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 222618461 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147329055 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 222638525 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145344130 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 367962589 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145344130 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84848081 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884040205543158e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚310699505011712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 222618461 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15526631317504 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚344994286632448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15526631317504 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15796532443648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15481866595006 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 367982655 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15526462304256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚329197778128896 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚311042873365182 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚295561006770176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147328532 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 _step‚905 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15796751403520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15526462304256 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚329197754188800!¹ý&’(÷M XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 _step‚905 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15796751403520 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15526462304256 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚521 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 242899968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚329197754188800 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 367962589 EDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚310699505011712 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚295560989730816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145344128 _runtime‚8645.191548358 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 367982655 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚326226136329216 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚84848083 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚310699487008768 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62480972 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145344128 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15481701473394 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 222618461 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147329055 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 222638525 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 145344130 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 367962589 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145344130 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚84848081 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 % _timestamp‚1.7884040205543158e+09 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚310699505011712 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 222618461 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15526631317504 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚344994286632448 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15526631317504 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15796532443648 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15481866595006 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 367982655 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15526462304256 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚329197778128896 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚311042873365182 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚295561006770176 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚523 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147328532 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872‰l‚Äã2“(- _wandbruntime‚8645  _runtime‚8645+„ ¹X”(Ò 2loss/SRqqt3/assignment_loss‚1.3333562277922673 .loss/FRt1/detection_loss‚2.372908046299761 /loss/FBt3/detection_loss‚2.6647109627723693 _runtime‚ 8645.19264631 2loss/SRqqt4/assignment_loss‚1.0148592338326476 1loss/FBt2/assignment_loss‚0.45798168420139973 1loss/SRqqt4/detection_loss‚1.2384784712546295 trainer/global_step‚10999 1loss/SRqqt1/detection_loss‚2.0406992366147594 % _timestamp‚1.7884040205558946e+09 0loss/FBt3/assignment_loss‚0.4391807457432151 /loss/FRt3/detection_loss‚2.5074543321833893 /loss/FBt4/detection_loss‚0.6639676389253039 .loss/FBt2/detection_loss‚2.941631919145584 &loss/total_loss‚43.259653666405924 epoch‚9 0loss/FRt2/assignment_loss‚3.0182548874900457 2loss/SRqqt2/assignment_loss‚1.7817211713109697 /loss/FBt1/detection_loss‚2.0433252283505032 2loss/SRqqt1/assignment_loss‚1.5094916794752313 1loss/FBt1/assignment_loss‚0.44741030829786155 1loss/SRqqt2/detection_loss‚2.3121730702263967 _step‚906 -loss/FRt4/assignment_loss‚1.5142984466675 -loss/FBt4/assignment_loss‚0.2427398491756 .loss/FRt2/detection_loss‚3.116592001347315 /loss/FRt4/detection_loss‚2.1283611059188843 1loss/SRqqt3/detection_loss‚1.6782299113859023 0loss/FRt1/assignment_loss‚3.0202834091403266 /loss/FRt3/assignment_loss‚2.771544098854065®3N¸]•(× 0loss/FBt3/assignment_loss‚0.4391807457432151 /loss/FRt3/detection_loss‚2.5074543321833893 /loss/FBt4/detection_loss‚0.6639676389253039 .loss/FBt2/detection_loss‚2.941631919145584 &loss/total_loss‚43.259653666405924 epoch‚9 0loss/FRt2/assignment_loss‚3.0182548874900457 2loss/SRqqt2/assignment_loss‚1.7817211713109697 /loss/FBt1/detection_loss‚2.0433252283505032 2loss/SRqqt1/assignment_loss‚1.5094916794752313 1loss/FBt1/assignment_loss‚0.44741030829786155 1loss/SRqqt2/detection_loss‚2.3121730702263967 _step‚906 -loss/FRt4/assignment_loss‚1.5142984466675 -loss/FBt4/assignment_loss‚0.2427398491756 .loss/FRt2/detection_loss‚3.116592001347315 /loss/FRt4/detection_loss‚2.1283611059188843 1loss/SRqqt3/detection_loss‚1.6782299113859023 0loss/FRt1/assignment_loss‚3.0202834091403266 /loss/FRt3/assignment_loss‚2.771544098854065 2loss/SRqqt3/assignment_loss‚1.3333562277922673 .loss/FRt1/detection_loss‚2.372908046299761 /loss/FBt3/detection_loss‚2.6647109627723693 _runtime‚ 8645.19264631 2loss/SRqqt4/assignment_loss‚1.0148592338326476 1loss/FBt2/assignment_loss‚0.45798168420139973 1loss/SRqqt4/detection_loss‚1.2384784712546295 trainer/global_step‚10999 1loss/SRqqt1/detection_loss‚2.0406992366147594 % _timestamp‚1.7884040205558946e+09ŠºV–(‚(:’ ¾ÂãÔ¹Ù ® gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚35.0 gpu.0.powerWatts‚145.354' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.070800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0êž;;Š—(‚(:ÿ ¾ÂãÔòû†± disk.rbd0.in‚ 0.0546875 disk.rbd0.out‚ 1.0986328125 disk.drbd1716.out‚0 network.sent‚ 2.7485146e+07* disk./.usagePercent‚15.849368843121203% disk.nvme0n1.out‚41226.9501953125 disk.sda2.out‚0 disk.dm-1.out‚ 8080.515625 disk.drbd2080.in‚0 disk.sda1.out‚0" disk.dm-0.out‚41226.9501953125 disk.nvme0c0n1.in‚ 416.734375 disk.sda.in‚ 0.06640625 disk.drbd2266.out‚0 disk.drbd2265.out‚0) proc.memory.percent‚16.44587516784668 cpu‚25.376870665304523% disk./.usageGB‚236.16704940795898 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda3.out‚ 8080.515625 proc.cpu.threads‚17' disk.nvme0c0n1.out‚41226.9501953125 disk.nvme0n1.in‚ 416.734375 disk.sda3.in‚ 0.06640625( proc.memory.availableMB‚ 394.55078125 disk.rbd1.out‚ 23.099609375 disk.drbd2265.in‚0 network.recv‚ 3.079757e+06 disk.rbd1.in‚0 disk.dm-1.in‚ 0.06640625 disk.dm-0.in‚ 416.734375 disk.sda1.in‚0$ memory_percent‚97.59185314178467" proc.memory.rssMB‚ 2694.4921875 disk.sda.out‚ 8080.515625 disk.sda2.in‚0 disk.drbd1716.in‚0×õë¼’˜(‚(:‡ ÍÂãÔܨˆ® gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚35.0 gpu.0.powerWatts‚144.287' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.8574 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¨÷ߤ™(‚(:„  ÍÂãÔÊãå°) proc.memory.percent‚16.44587516784668 network.sent‚ 2.7594055e+07 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd1.out‚ 23.267578125 disk.nvme0c0n1.in‚ 416.734375 network.recv‚ 3.086586e+06 disk.sda3.in‚ 0.06640625 cpu‚25.376279680541266* disk./.usagePercent‚15.849367307075998 disk.dm-1.in‚ 0.06640625 disk.rbd1.in‚0( disk.nvme0c0n1.out‚41227.47802734375 disk.nvme0n1.in‚ 416.734375 disk.sda2.in‚0 disk.drbd1716.in‚0" proc.memory.rssMB‚ 2694.4921875$ disk./.usageGB‚236.1670265197754 disk.dm-1.out‚ 8080.9765625 disk.drbd2080.in‚0 disk.sda.out‚ 8080.9765625 disk.drbd1716.out‚0# disk.dm-0.out‚41227.47802734375 disk.sda1.in‚0 disk.sda.in‚ 0.06640625$ memory_percent‚97.59032726287842 disk.sda1.out‚0 proc.cpu.threads‚17 disk.rbd0.out‚ 1.0986328125 disk.dm-0.in‚ 416.734375& disk.nvme0n1.out‚41227.47802734375 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.rbd0.in‚ 0.0546875 disk.sda3.out‚ 8080.9765625( proc.memory.availableMB‚ 394.80078125`ÈG2š(- _wandbruntime‚8679  _runtime‚8679œí /È›( _runtime‚8679.871638902 _step‚907 & lr-AdamW/pg1‚4.818847219562099e-06 & lr-AdamW/pg2‚4.818847219562099e-06 trainer/global_step‚11049 % _timestamp‚1.7884040552347977e+09QšÁ<Íœ(Ç & lr-AdamW/pg1‚4.818847219562099e-06 & lr-AdamW/pg2‚4.818847219562099e-06 trainer/global_step‚11049 % _timestamp‚1.7884040552347977e+09 _runtime‚8679.871638902 _step‚907‹z¡¢]2(- _wandbruntime‚8679  _runtime‚8679y#cŽv ž(çO N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15549784127258 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145964419 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85209556 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚346480492466688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚327630613372416 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145964421 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚312381989792538 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚346480724894208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚327630613372416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚312035703883776 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚312382171953302 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚296832222704640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85209554 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 147955636 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15866003814912 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 369555795 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚330614512591872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚327630426369536 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145964419 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚330614488651776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15549949248662 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15594740488704 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 $ _timestamp‚1.788404055235496e+09 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15594909488640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15594740488704 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15594909488640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15866212302336 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 _step‚908 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 223611414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 223591376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62745537 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =6DeviceStatsMonitoŠLÄÔwr.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145964421 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62746080 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚312035685880832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 147955091 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚312035703883776 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚312035685880832 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚11049 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520¬¯çò'Ÿ(ìO [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15549784127258 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 145964419 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚85209556 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚346480492466688 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚327630613372416 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 145964421 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚312381989792538 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚346480724894208 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚327630613372416 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚312035703883776 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚312382171953302 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚296832222704640 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85209554 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 147955636 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15866003814912 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 369555795 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚330614512591872 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚327630426369536 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 145964419 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚330614488651776 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15549949248662 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15594740488704 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 $ _timestamp‚1.788404055235496e+09 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15594909488640 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚545 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15594740488704 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15594909488640 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15866212302336 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 _step‚908 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 223611414 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 223591376 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚62745537 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 145964421 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚62746080 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚312035685880832 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 147955091 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚312035703883776 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚312035685880832 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 trainer/global_step‚11049 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408Œñ²2 (- _wandbruntime‚8680  _runtime‚8680AÆ!÷&¡(ñM QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚296858148847616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚346510824795136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚330643401600000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚312409504300390 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 369588241 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 223631227 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚312062973248512 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 trainer/global_step‚11049 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85216933 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145977080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62751377 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 369608307 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚346511057209344 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145977078 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62750865 _step‚909 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 223611163 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 369608307 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147967796 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15867631669248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚330643425540096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145977078 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15596133626880 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147968310 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 223631227 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚327659088872448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 % _timestamp‚1.7884040559037812e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚312062955245568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚312062955245568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944&ž„€ü&¢(öM QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 trainer/global_step‚11049 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85216933 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 145977080 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚62751377 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 369608307 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚346511057209344 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 145977078 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚62750865 _step‚909 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 223611163 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 369608307 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 147967796 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚15867631669248 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚330643425540096 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 145977078 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15596133626880 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 147968310 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 223631227 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚327659088872448 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚512 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 % _timestamp‚1.7884040559037812e+09 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚312062955245568 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚312062955245568 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚296858148847616 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚346510824795136 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚330643401600000 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚312409504300390 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 369588241 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 223631227 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚312062973248512 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128û†+Q2£(- _wandbruntime‚8680  _runtime‚8680.PßÁ¤(Õ .loss/FBt2/detection_loss‚2.371546681722005 0loss/FRt3/assignment_loss‚2.7400488486656775 2loss/SRqqt1/assignment_loss‚1.1016067095586788 2loss/SRqqt3/assignment_loss‚1.4639507107070235 -loss/FRt3/detection_loss‚2.18393307322493 .loss/FBt4/detection_loss‚0.766360248439014 _step‚910 0loss/FRt1/assignment_loss‚3.0647648952224036 0loss/FBt4/assignment_loss‚0.1439070281096367 /loss/FRt4/detection_loss‚1.8901782006673191 1loss/SRqqt1/detection_loss‚2.1185095034863637 &loss/ˆE½Ltotal_loss‚43.719033106238655 epoch‚9 % _timestamp‚1.7884040559054773e+09 0loss/FBt1/assignment_loss‚0.3310033012140347 /loss/FRt1/detection_loss‚2.5263370268724183 1loss/SRqqt3/detection_loss‚1.5548898851285216 .loss/FRt2/detection_loss‚3.770652911067009 1loss/SRqqt2/assignment_loss‚2.398890597417074 1loss/SRqqt4/assignment_loss‚1.193443716069482 0loss/FBt2/assignment_loss‚0.2258704088316487 /loss/FBt3/detection_loss‚2.7784119963645937 _runtime‚8680.542225545 0loss/FRt2/assignment_loss‚3.8420867800712584 1loss/FBt3/assignment_loss‚0.25245140194310806 0loss/FRt4/assignment_loss‚1.5188390794007673 1loss/SRqqt2/detection_loss‚2.2302474204231713 1loss/SRqqt4/detection_loss‚1.0092449244615194 /loss/FBt1/detection_loss‚2.2418577571709952 trainer/global_step‚11049Î"9)`¥(Ú 2loss/SRqqt1/assignment_loss‚1.1016067095586788 2loss/SRqqt3/assignment_loss‚1.4639507107070235 -loss/FRt3/detection_loss‚2.18393307322493 .loss/FBt4/detection_loss‚0.766360248439014 _step‚910 0loss/FRt1/assignment_loss‚3.0647648952224036 0loss/FBt4/assignment_loss‚0.1439070281096367 /loss/FRt4/detection_loss‚1.8901782006673191 1loss/SRqqt1/detection_loss‚2.1185095034863637 &loss/total_loss‚43.719033106238655 epoch‚9 % _timestamp‚1.7884040559054773e+09 0loss/FBt1/assignment_loss‚0.3310033012140347 /loss/FRt1/detection_loss‚2.5263370268724183 1loss/SRqqt3/detection_loss‚1.5548898851285216 .loss/FRt2/detection_loss‚3.770652911067009 1loss/SRqqt2/assignment_loss‚2.398890597417074 1loss/SRqqt4/assignment_loss‚1.193443716069482 0loss/FBt2/assignment_loss‚0.2258704088316487 /loss/FBt3/detection_loss‚2.7784119963645937 _runtime‚8680.542225545 0loss/FRt2/assignment_loss‚3.8420867800712584 1loss/FBt3/assignment_loss‚0.25245140194310806 0loss/FRt4/assignment_loss‚1.5188390794007673 1loss/SRqqt2/detection_loss‚2.2302474204231713 1loss/SRqqt4/detection_loss‚1.0092449244615194 /loss/FBt1/detection_loss‚2.2418577571709952 trainer/global_step‚11049 .loss/FBt2/detection_loss‚2.371546681722005 0loss/FRt3/assignment_loss‚2.7400488486656775Žv„¦(‚(:’ ÜÂãÔÏÉ„® gpu.0.gpu‚40.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚114.196' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚22.839199999999998 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0©Øy’§(‚(:‡  ÜÂãÔ¯Âã° disk.dm-0.in‚ 416.734375 network.sent‚ 2.7600049e+07 disk.rbd1.in‚0 disk.rbd1.out‚ 23.341796875( proc.memory.availableMB‚ 393.98828125* disk./.usagePercent‚15.760330190735033 disk.sda2.out‚0 disk.drbd2080.in‚0 cpu‚25.375717820037607 proc.cpu.threads‚17 disk.sda.in‚ 0.06640625 disk.nvme0c0n1.in‚ 416.734375 disk.dm-1.in‚ 0.06640625 disk.rbd0.out‚ 1.0986328125 disk.sda3.out‚ 8081.92578125 disk.drbd1716.in‚0# disk.dm-0.out‚41228.13037109375 disk.rbd0.in‚ 0.0546875 disk.sda2.in‚0 network.recv‚ 3.090405e+06& disk.nvme0n1.out‚41228.13037109375 disk.dm-1.out‚ 8081.92578125 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda.out‚ 8081.92578125 disk.drbd1716.out‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda3.in‚ 0.06640625$ memory_percent‚97.59528636932373) proc.memory.percent‚16.44587516784668$ disk./.usageGB‚234.8403091430664 disk.nvme0n1.in‚ 416.734375 disk.drbd2266.in‚0" proc.memory.rssMB‚ 2694.4921875( disk.nvme0c0n1.out‚41228.13037109375 disk.sda1.out‚0¡ôJ¨(‚(:’ ëÂãÔ–×°® gpu.0.gpu‚15.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚141.047' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.209400000000002 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ˆfÅÿˆ©(‚(:ý ëÂãÔ°·‚² disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.sda3.out‚ 8082.96875) proc.memory.percent‚16.44587516784668 disk.sda2.out‚0 disk.rbd0.out‚ 1.0986328125 proc.cpu.threads‚17 network.recv‚ 3.096837e+06 disk.drbd1716.in‚0 disk.dm-1.out‚ 8082.96875 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.sda3.in‚ 0.06640625 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 416.734375 network.sent‚ 2.7708886e+07$ disk./.usageGB‚234.8403091430664 disk.sda.in‚ 0.06640625 disk.sda2.in‚0 disk.drbd2080.in‚0# disk.dm-0.out‚41229.08935546875 disk.nvme0n1.in‚ 416.734375 disk.dm-1.in‚ 0.06640625 disk.sda1.in‚0 disk.sda1.out‚0 disk.rbd1.out‚ 23.404296875" proc.memory.rssMB‚ 2694.4921875 disk.drbd2265.in‚0' proc.memory.availableMB‚ 393.3671875 cpu‚25.375150350229728 disk.drbd2265.out‚0 disk.dm-0.in‚ 416.734375( disk.nvme0c0n1.out‚41229.08935546875& disk.nvme0n1.out‚41229.08935546875$ memory_percent‚97.59907722473145* disk./.usagePercent‚15.760330190735033 disk.sda.out‚ 8082.96875 disk.rbd0.in‚ 0.0546875¯k¿ª(‚(:… úÂãԥኮ gpu.0.gpu‚37.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚146.42' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.284 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0u·¬…«(‚(:ú úÂãÔêåñ° disk.rbd0.out‚ 1.0986328125 disk.nvme0c0n1.in‚ 416.734375 cpu‚25.37457048921708% disk./.usageGB‚234.84029006958008 disk.sda1.in‚0$ memory_percent‚97.59485721588135 network.recv‚ 3.100309e+06 disk.sda1.out‚0 disk.dm-1.in‚ 0.06640625 disk.dm-1.out‚ 8083.4765625) proc.memory.percent‚16.44587516784668 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.dm-0.out‚41229.80078125 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.rbd0.in‚ 0.0546875 disk.sda3.in‚ 0.06640625 disk.rbd1.out‚ 23.404296875% disk.nvme0c0n1.out‚41229.80078125 proc.cpu.threads‚17 disk.drbd2080.out‚0 disk.rbd1.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 394.05859375" proc.memory.rssMB‚ 2694.4921875 network.sent‚ 2.771432e+07# disk.nvme0n1.out‚41229.80078125 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2265.in‚0 disk.dm-0.in‚ 416.734375 disk.sda3.out‚ 8083.4765625 disk.nvme0n1.in‚ 416.734375 disk.sda.in‚ 0.06640625 disk.sda.out‚ 8083.4765625 disk.sda2.out‚0* disk./.usagePercent‚15.760328910697364Ø1FÌ2¬(- _wandbruntime‚8715  _runtime‚8715U4¿Ä­(¾ % lr-AdamW/pg2‚4.37014621888562e-06 trainer/global_step‚11099 $ _timestamp‚1.788404090810913e+09 _runtime‚ 8715.44777643 _step‚911 % lr-AdamW/pg1‚4.37014621888562e-06jWá É®(à % lr-AdamW/pg2‚4.37014621888562e-06 trainer/global_step‚11099 $ _timestamp‚1.788404090810913e+09 _runtime‚ 8715.44777643 _step‚911 % lr-AdamW/pg1‚4.37014621888562e-06¬F߆2¯(- _wandbruntime‚8715  _runtime‚8715à]åí'°(çO @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 224584078 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚524 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329063565928960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15664580811264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚347997020612096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 371181447 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚313399172120576 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313399172120576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15936884549120 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚329063752931840 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15619421067070 _runtime‚ 8715.44874632 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚313399154117632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15664411811328 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 148594719 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 224584078 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 224604116 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15937095133696 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85578404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329063565928960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15664580811264 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚347997255136768 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚332060160003072 % _timestamp‚1.7884040908116612e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146597371 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15664411811328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚298129364782080 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15619255945666 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 371201487 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146597369 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313399154117632 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚329063752931840 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63016315 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146597369 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63016837 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚298129381821440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936‡nC*ò'±(ìO R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚329063565928960 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15664580811264 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚347997255136768 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚332060160003072 % _timestamp‚1.7884040908116612e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 146597371 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15664411811328 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚298129364782080 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15619255945666 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 371201487 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 146597369 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚313399154117632 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚329063752931840 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63016315 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 146597369 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63016837 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚298129381821440 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 224584078 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚524 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚329063565928960 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15664580811264 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚522 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚347997020612096 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 371181447 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚313399172120576 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚313399172120576 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚15936884549120 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚329063752931840 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15619421067070 _runtime‚ 8715.44874632 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚313399154117632 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15664411811328 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 148594719 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 224584078 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 224604116 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚15937095133696 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚85578404 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148607372 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15938298586112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 371213893 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85585783 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329092415448064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15665804949504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146610028 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15665973962752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 224603865 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298155307964416 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884040914856546e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚348027347597312 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 371233959 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 371213893 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚348027584205824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚913 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _runtime‚8716.122943769 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 224603865 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146610030 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 trainer/global_step‚11099 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 148607897 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146610028 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15620810231822 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚07Ńý&´(÷M K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 224603865 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 146610030 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 trainer/global_step‚11099 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 148607897 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 146610028 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 212668416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15620810231822 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚313426441485312 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 224623929 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚329092415448064 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚313776135235598 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 224623929 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15665973962752 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63022114 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚313775953074626 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 148607372 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚15938298586112 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 371213893 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚85585783 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚329092415448064 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15665804949504 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 146610028 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15665973962752 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 224603865 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚298155307964416 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884040914856546e+09 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚348027347597312 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 371233959 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 371213893 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚348027584205824 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _step‚913 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _runtime‚8716.122943769‘سæL2µ(- _wandbruntime‚8716  _runtime‚8716¹qoÂb¶(Ü 0loss/FRt1/assignment_loss‚2.6678662832747113 /loss/FBt4/detection_loss‚0.9025618314138434 /loss/FRt4/detection_loss‚1.9809734532166094 0loss/FRt4/assignment_loss‚1.4983690551349096 2loss/SRqqt2/assignment_loss‚1.8577649811548846 2loss/SRqqt3/assignment_loss‚1.3648508397362582 3loss/FBt3/assignment_loss‚0.0036492359358586413 /loss/FRt3/detection_loss‚2.5472038349038675 1loss/SRqqt2/detection_loss‚2.5938811791794643 1loss/SRqqt3/detection_loss‚2.0766951594410874 -loss/FBt3/detection_loss‚4.21936160326004 % _timestamp‚1.7884040914873102e+09 1loss/FBt4/assignment_loss‚0.11769467844936249 -loss/FBt2/detection_loss‚2.55911540443247 &loss/total_loss‚46.840740684808786 2loss/SRqqt1/assignment_loss‚1.5698955574266191 /loss/FRt1/detection_loss‚2.3516019192781856 1loss/SRqqt4/detection_loss‚0.9629285735565991 /loss/FRt2/assignment_loss‚3.775926322772585 2loss/SRqqt4/assignment_loss‚1.0990657714151202 1loss/FBt2/assignment_loss‚0.44781835741266335 /loss/FRt2/detection_loss‚3.0160480737686157 trainer/global_step‚11099 0loss/SRqqt1/detection_loss‚1.872550773733067 epoch‚9 _step‚914 0loss/FRt3/assignment_loss‚3.2150643812982658 3loss/FBt1/assignment_loss‚0.0024647183405287856 .loss/FBt1/detection_loss‚4.137388696273168 _runtime‚8716.124073029ÙÞg·(á 2loss/SRqqt1/assignment_loss‚1.5698955574266191 /loss/FRt1/detection_loss‚2.3516019192781856 1loss/SRqqt4/detection_loss‚0.9629285735565991 /loss/FRt2/assignment_loss‚3.775926322772585 2loss/SRqqt4/assignment_loss‚1.0990657714151202 1loss/FBt2/assignment_loss‚0.44781835741266335 /loss/FRt2/detection_loss‚3.0160480737686157 trainer/global_step‚11099 0loss/SRqqt1/detection_loss‚1.872550773733067 epoch‚9 _step‚914 0loss/FRt3/assignment_loss‚3.2150643812982658 3loss/FBt1/assignment_loss‚0.0024647183405287856 .loss/FBt1/detection_loss‚4.137388696273168 _runtime‚8716.124073029 0loss/FRt1/assignment_loss‚2.6678662832747113 /loss/FBt4/detection_loss‚0.9025618314138434 /loss/FRt4/detection_loss‚1.9809734532166094 0loss/FRt4/assignment_loss‚1.4983690551349096 2loss/SRqqt2/assignment_loss‚1.8577649811548846 2loss/SRqqt3/assignment_loss‚1.3648508397362582 3loss/FBt3/assignment_loss‚0.0036492359358586413 /loss/FRt3/detection_loss‚2.5472038349038675 1loss/SRqqt2/detection_loss‚2.5938811791794643 1loss/SRqqt3/detection_loss‚2.0766951594410874 -loss/FBt3/detection_loss‚4.21936160326004 % _timestamp‚1.7884040914873102e+09 1loss/FBt4/assignment_loss‚0.11769467844936249 -loss/FBt2/detection_loss‚2.55911540443247 &loss/total_loss‚46.840740684808786’Ûq“˜‘¸(‚(:† ‰ÃãÔ’ꈮ gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚139.155' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.831 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Úy¥ø‘¹(‚(:†  ‰ÃãÔþ™ß°" proc.memory.rssMB‚ 2694.4921875 disk.drbd2265.in‚0 disk.drbd2080.in‚0# disk.dm-0.out‚41232.66845703125 disk.sda1.in‚0( disk.nvme0c0n1.out‚41232.66845703125 disk.sda.in‚ 0.06640625 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 416.734375 disk.rbd1.out‚ 23.583984375 disk.nvme0n1.in‚ 416.734375 disk.rbd1.in‚0$ memory_percent‚97.59633541107178 cpu‚25.37404382940676 disk.drbd1716.out‚0( proc.memory.availableMB‚ 393.81640625) proc.memory.percent‚16.44587516784668 network.sent‚ 2.7720271e+07 disk.dm-1.in‚ 0.06640625 disk.sda3.out‚ 8084.01171875 disk.nvme0c0n1.in‚ 416.734375 disk.sda.out‚ 8084.01171875 disk.sda2.out‚0 disk.drbd1716.in‚0& disk.nvme0n1.out‚41232.66845703125 disk.sda2.in‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.rbd0.out‚ 1.0986328125 disk.sda3.in‚ 0.06640625 proc.cpu.threads‚17 network.recv‚ 3.104696e+06) disk./.usagePercent‚15.76033531088572% disk./.usageGB‚234.84038543701172 disk.dm-1.out‚ 8084.01171875 disk.rbd0.in‚ 0.0546875 disk.sda1.out‚0ÀÒÓ’º(‚(:‡ ˜ÃãÔûÙª® gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚135.619' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.1238 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ãÍ«¥Œ»(‚(:  ˜ÃãÔõ‘’± disk.rbd0.out‚ 1.0986328125 disk.rbd1.in‚0 disk.sda.out‚ 8084.6171875 proc.cpu.threads‚17 network.sent‚ 2.7828735e+07$ disk./.usageGB‚234.8402862548828 disk.nvme0n1.in‚ 416.734375 disk.sda2.in‚0 disk.drbd1716.in‚0" proc.memory.rssMB‚ 2694.4921875 network.recv‚ 3.110881e+06 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.06640625 disk.rbd1.out‚ 23.646484375 disk.drbd2266.in‚0# memory_percent‚97.5947380065918) proc.memory.percent‚16.44587516784668 disk.dm-0.in‚ 416.734375 disk.nvme0c0n1.in‚ 416.734375 disk.drbd1716.out‚0 cpu‚25.373458667844705* disk./.usagePercent‚15.760328654689829 disk.drbd2265.in‚0 disk.sda2.out‚0& proc.memory.availableMB‚ 394.078125 disk.drbd2080.in‚0# disk.dm-0.out‚41232.89111328125 disk.dm-1.in‚ 0.06640625 disk.sda3.out‚ 8084.6171875& disk.nvme0n1.out‚41232.89111328125 disk.sda.in‚ 0.06640625 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.rbd0.in‚ 0.0546875 disk.sda1.in‚0( disk.nvme0c0n1.out‚41232.89111328125 disk.dm-1.out‚ 8084.6171875ŠŒy2¼(- _wandbruntime‚8750  _runtime‚8750\í¾„Æ½(À % lr-AdamW/pg1‚3.94293593031375e-06 % lr-AdamW/pg2‚3.94293593031375e-06 trainer/global_step‚11149 % _timestamp‚1.7884041262764447e+09 _runtime‚8750.913289534 _step‚915í¼úy˾(Å % lr-AdamW/pg1‚3.94293593031375e-06 % lr-AdamW/pg2‚3.94293593031375e-06 trainer/global_step‚11149 % _timestamp‚1.7884041262764447e+09 _runtime‚8750.913289534 _step‚915“þ32¿(- _wandbruntime‚8750  _runtime‚8750äˆv`î'À(èO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 225596818 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16007825216000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚330496892491264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚314762640357376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚315115251662954 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚314762622354432 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147230321 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 225596818 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 % _timestamp‚1.7884041262771635e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15734083133952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚330496705488384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚333505783474176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚314762640357376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚314762622354432 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚349513608690176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚299426523898880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15688892885478 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚556 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 trainer/global_step‚11149 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15688727764074 _step‚916 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚349513841117696 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚299426540938240 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147230319 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 149232660 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63285406 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚315115433823718 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 225576780 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚330496705488384 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416Šá½ýÁ(íO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 225576780 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚330496705488384 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allhk?ö$ocated‚ 225596818 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16007825216000 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚330496892491264 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚314762640357376 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚315115251662954 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚314762622354432 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147230321 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 225596818 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 % _timestamp‚1.7884041262771635e+09 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15734083133952 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚330496705488384 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚333505783474176 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚314762640357376 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚314762622354432 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚349513608690176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚299426523898880 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15688892885478 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚556 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 trainer/global_step‚11149 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15688727764074 _step‚916 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚349513841117696 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚299426540938240 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 147230319 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 149232660 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63285406 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚315115433823718 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0”]™ä"2Â(- _wandbruntime‚8751  _runtime‚8751Ü6•ïø&Ã(òM D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 372839545 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚8751.583922606 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚314789909722112 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15735476272128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85954631 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚299452467081216 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15735645285376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226122752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚299452484120576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 225596567 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚506 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚333534720362496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16009453166592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202182656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149245690 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63291565 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16009250983936 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚315142584009834 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚330525555007488 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 372859611 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚508 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚333534696422400 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15735476272128 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147242980 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 372839545 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884041269466343e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 372859611 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚330525367991296 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 149246198 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 225616631 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15690116928618 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 trainer/global_step‚11149 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚314789891719168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚330525555007488 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15735645285376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257d!ÑÀý&Ä(÷M SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 225616631 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 225596567 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚349543947406336 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚299452467081216 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 E6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15735645285376 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 226122752 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚299452484120576 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 225596567 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚506 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚333534720362496 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16009453166592 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 202182656 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149245690 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63291565 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16009250983936 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚315142584009834 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚330525555007488 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 372859611 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚508 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚333534696422400 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15735476272128 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147242980 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 372839545 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 % _timestamp‚1.7884041269466343e+09 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 372859611 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚330525367991296 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 149246198 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 225616631 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15690116928618 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 trainer/global_step‚11149 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚314789891719168 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚330525555007488 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15735645285376 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 372839545 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 _runtime‚8751.583922606 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚314789909722112 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15735476272128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚85954631 P2Ï(- _wandbruntime‚8786  _runtime‚8786{0 ì'Ð(æO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15803754456576 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86316106 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚316126090591232 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16078864045568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15803923456512 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 149873107 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 149872553 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15758199582482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚334951430885376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚316481882598162 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 374432751 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 226569482 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15803923456512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _step‚920 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚351030318871040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚331929845047808 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 226589520 _runtime‚8786.654281016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 374452791 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚331929845047808 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚331930032050688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚316126090591232 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚316126108594176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚300723683015680 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚316482064758926 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147863271 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 226589520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147863271 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63557001 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16078651363840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147863269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63556449 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15758364703886 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15803754456576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚351030082249216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360…Ztóñ'Ñ(ëO \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15803923456512 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 149873107 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 149872553 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15758199582482 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚334951430885376 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚316481882598162 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 374432751 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 226569482 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15803923456512 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _step‚920 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚351030318871040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚331929845047808 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 226589520 _runtime‚8786.654281016 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 374452791 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚331929845047808 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚331930032050688 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚316126090591232 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚316126108594176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚300723683015680 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚316482064758926 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 147863271 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 226589520 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 147863271 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63557001 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16078651363840 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 147863269 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63556449 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15758364703886 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15803754456576 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚351030082249216 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15803754456576 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86316106 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚316126090591232 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16078864045568 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992˜ÞdÊ2Ò(- _wandbruntime‚8787  _runtime‚87876uôÔ#Ó(òM REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63561732 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 226589269 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚351060416918528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 226589269 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 226609333 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚300749626198016 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚331958694566912 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚316153377958912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15759753868638 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 trainer/global_step‚11199 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚316153359955968 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149885213 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚300749643237376 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147875928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86323481 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147875930 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚351060649332736 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 226609333 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚316153377958912 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15805147594752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15805316608000 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 149885763 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚548 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63562280 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚316509397106014 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 374465197 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 _step‚921 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147875928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15805316608000 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147875930 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15759588747026 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚550 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16080281559040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.pe…fàak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚316509214945042 _runtime‚8787.334736913 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86323483 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 374465197 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374485263 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0{uGÓý&Ô(÷M PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 147875928 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15805316608000 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 147875930 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15759588747026 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚550 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16080281559040 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚316509214945042 _runtime‚8787.334736913 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86323483 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 374465197 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 BDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 374485263 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚63561732 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 226589269 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚351060416918528 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 226589269 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 226609333 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚300749626198016 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚331958694566912 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚316153377958912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15759753868638 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 trainer/global_step‚11199 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚316153359955968 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 149885213 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚300749643237376 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 147875928 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86323481 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 147875930 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚351060649332736 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 226609333 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚316153377958912 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15805147594752 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15805316608000 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 149885763 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚548 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63562280 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚316509397106014 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 374465197 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 _step‚921 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217™%LÐ2Õ(- _wandbruntime‚8787  _runtime‚8787–)Û(^Ö(Ø % _timestamp‚1.7884041626990385e+09 0loss/FRt3/assignment_loss‚2.7740275152027607 1loss/SRqqt3/assignment_loss‚1.478330913577871 .loss/FRt1/detection_loss‚2.228027719928294 .loss/FRt3/detection_loss‚2.259188713505864 1loss/SRqqt1/detection_loss‚1.7719568344099181 1loss/SRqqt2/detection_loss‚2.4209122402327403 epoch‚9 2loss/SRqqt1/assignment_loss‚1.5014766611691033 2loss/SRqqt2/assignment_loss‚1.9509103078927312 1loss/FBt4/assignment_loss‚0.23574658426019213 1loss/SRqqt3/detection_loss‚1.5324255189505116 0loss/FBt1/assignment_loss‚0.3750880803545442 1loss/FBt2/assignment_loss‚0.12821194876331296 0loss/FBt3/assignment_loss‚0.5054201309879621 /loss/FRt4/detection_loss‚1.7815821160163199 /loss/FBt2/detection_loss‚3.4846217466725244 /loss/FBt4/detection_loss‚0.7361808692494148 _runtime‚8787.335753441 0loss/FRt2/assignment_loss‚3.8397595268838547 /loss/FBt3/detection_loss‚2.4734710454940796 %loss/total_loss‚42.93307078556746 _step‚922 trainer/global_step‚11199 1loss/SRqqt4/detection_loss‚0.9715276695787907 2loss/SRqqt4/assignment_loss‚1.1545664677899834 .loss/FBt1/detection_loss‚2.398300774395466 0loss/FRt1/assignment_loss‚2.4271400546541018 0loss/FRt4/assignment_loss‚1.7842630731816194 /loss/FRt2/detection_loss‚2.7199342724154976â_½¼c×(Ý % _timestamp‚1.7884041626990385e+09 0loss/FRt3/assignment_loss‚2.7740275152027607 1loss/SRqqt3/assignment_loss‚1.478330913577871 .loss/FRt1/detection_loss‚2.228027719928294 .loss/FRt3/detection_loss‚2.259188713505864 1loss/SRqqt1/detection_loss‚1.7719568344099181 1loss/SRqqt2/detection_loss‚2.4209122402327403 epoch‚9 2loss/SRqqt1/assignment_loss‚1.5014766611691033 2loss/SRqqt2/assignment_loss‚1.9509103078927312 1loss/FBt4/assignment_loss‚0.23574658426019213 1loss/SRqqt3/detection_loss‚1.5324255189505116 0loss/FBt1/assignment_loss‚0.3750880803545442 1loss/FBt2/assignment_loss‚0.12821194876331296 0loss/FBt3/assignment_loss‚0.5054201309879621 /loss/FRt4/detection_loss‚1.7815821160163199 /loss/FBt2/detection_loss‚3.4846217466725244 /loss/FBt4/detection_loss‚0.7361808692494148 _runtime‚8787.335753441 0loss/FRt2/assignment_loss‚3.8397595268838547 /loss/FBt3/detection_loss‚2.4734710454940796 %loss/total_loss‚42.93307078556746 _step‚922 trainer/global_step‚11199 1loss/SRqqt4/detection_loss‚0.9715276695787907 2loss/SRqqt4/assignment_loss‚1.1545664677899834 .loss/FBt1/detection_loss‚2.398300774395466 0loss/FRt1/assignment_loss‚2.4271400546541018 0loss/FRt4/assignment_loss‚1.7842630731816194 /loss/FRt2/detection_loss‚2.7199342724154976š½ÁÃfØ(‚(:’ ÅÃãÔü¦® gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚146.131' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.226200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ƒ%Ó\Ù(‚(:„  ÅÃãÔÛ€õ° disk.nvme0n1.in‚ 416.734375& disk.nvme0n1.out‚41234.69287109375 disk.dm-1.in‚ 0.06640625 disk.dm-1.out‚ 8086.40234375 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 network.sent‚ 2.7949237e+07) disk./.usagePercent‚15.76033838297613 disk.rbd1.out‚ 23.8662109375 disk.dm-0.in‚ 416.734375 disk.sda3.in‚ 0.06640625 disk.sda2.in‚0% proc.memory.availableMB‚ 393.46875# disk.dm-0.out‚41234.69287109375 disk.sda.in‚ 0.06640625 disk.sda.out‚ 8086.40234375 disk.rbd0.out‚ 1.0986328125$ memory_percent‚97.59845733642578 disk.sda1.out‚0 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.rbd0.in‚ 0.0546875" proc.memory.rssMB‚ 2694.4921875 cpu‚25.371808809307066 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 416.734375 disk.drbd1716.in‚0) proc.memory.percent‚16.44587516784668 network.recv‚ 3.125432e+06$ disk./.usageGB‚234.8404312133789 disk.sda3.out‚ 8086.40234375( disk.nvme0c0n1.out‚41234.69287109375 disk.sda2.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 proc.cpu.threads‚17»SÀ„’Ú(‚(:‡ ÔÃãÔöÅ€® gpu.0.gpu‚33.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚35.0 gpu.0.powerWatts‚149.668' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.9336 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0é"ŒÛ(‚(:  ÔÃãÔ‚Ó¼° network.sent‚ 2.8057652e+07 disk.nvme0n1.in‚ 416.734375 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0$ memory_percent‚97.59712219238281 cpu‚25.371236464746797( disk.nvme0c0n1.out‚41235.06982421875 disk.drbd1716.out‚0) proc.memory.percent‚16.44587516784668 disk.dm-1.out‚ 8086.8984375 disk.drbd2080.out‚0 network.recv‚ 3.131318e+06 disk.rbd1.in‚0 disk.sda.out‚ 8086.8984375 disk.drbd2265.out‚0 disk.rbd0.out‚ 1.0986328125 disk.sda1.in‚0 disk.sda3.in‚ 0.06640625 disk.sda3.out‚ 8086.8984375% disk./.usageGB‚234.84041213989258 disk.rbd1.out‚ 23.9287109375 disk.dm-1.in‚ 0.06640625 disk.drbd2266.in‚0$ proc.memory.availableMB‚393.6875 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 416.734375 disk.sda2.out‚0 disk.dm-0.in‚ 416.734375& disk.nvme0n1.out‚41235.06982421875 disk.sda.in‚ 0.06640625 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.sda1.out‚0" proc.memory.rssMB‚ 2694.4921875) disk./.usagePercent‚15.76033710293846 disk.rbd0.in‚ 0.0546875# disk.dm-0.out‚41235.06982421875Òá»’Ü(‚(:‡ ãÃãԤɇ® gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚121.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚24.2658 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ž²û‰Ý(‚(:þ ãÃãÔ‡‘ï° disk.rbd0.in‚ 0.0546875' proc.memory.availableMB‚ 393.1953125 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.dm-0.in‚ 416.734375! disk.dm-0.out‚41235.662109375 disk.sda1.in‚0 disk.dm-1.out‚ 8087.3359375 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.sda2.in‚0 network.sent‚ 2.8063926e+07) disk./.usagePercent‚15.76033710293846 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.rbd1.out‚ 23.9287109375$ disk.nvme0n1.out‚41235.662109375$ memory_percent‚97.60012626647949 disk.drbd2265.out‚0 disk.sda2.out‚0 network.recv‚ 3.135878e+06 disk.nvme0c0n1.in‚ 416.734375 disk.sda.in‚ 0.06640625" proc.memory.rssMB‚ 2694.4921875 disk.sda.out‚ 8087.3359375 disk.drbd1716.in‚0% disk./.usageGB‚234.84041213989258 disk.drbd2265.in‚0 disk.rbd1.in‚0& disk.nvme0c0n1.out‚41235.662109375) proc.memory.percent‚16.44587516784668 disk.sda3.in‚ 0.06640625 disk.nvme0n1.in‚ 416.734375 cpu‚25.370689310616914 disk.rbd0.out‚ 1.0986328125 disk.sda3.out‚ 8087.3359375 disk.dm-1.in‚ 0.06640625 proc.cpu.threads‚17#ÍŒ2Þ(- _wandbruntime‚8822  _runtime‚8822F—\8Èß( _runtime‚8822.164475961 _step‚923 & lr-AdamW/pg1‚3.153339112233229e-06 & lr-AdamW/pg2‚3.153339112233229e-06 trainer/global_step‚11249 % _timestamp‚1.7884041975275984e+09B]=Íà(Ç _step‚923 & lr-AdamW/pg1‚3.153339112233229e-06 & lr-AdamW/pg2‚3.153339112233229e-06 trainer/global_step‚11249 % _timestamp‚1.7884041975275984e+09 _runtime‚8822.164475961›³E2á(- _wandbruntime‚8822  _runtime‚8822Ú^øî'â(èO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15873594779136 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚352546805368320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 376058403 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 150511263 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚317489558828032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336397078296576 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚333363171610112 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚8822.165367914 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚317848695694134 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚302020842132480 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333362984607232 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15873425779200 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚352546572940800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15873594779136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 376078443 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16149494644224 % _timestamp‚1.7884041975283334e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚333362984607232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _step‚924 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 227562184 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148496221 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86684956 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚333363171610112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚317489558828032 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63826309 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16149703131648 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚11249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 148496219 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86684954 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63826823 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 227582222 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚516 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚317489576830976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚514 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚317848513533370 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚333362984607232 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15873425779200 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚352546572940800 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15873594779136 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 376078443 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16149494644224 % _timestamp‚1.7884041975283334e+09 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚333362984607232 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 _step‚924 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 227562184 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 148496221 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚86684956 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚333363171610112 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚317489558828032 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚63826309 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16149703131648 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 trainer/global_step‚11249 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 148496219 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚86684954 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚63826823 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 227582222 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚516 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚317489576830976 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚514 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚317848513533370 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚352546805368320 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 376058403 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 150511263 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚317489558828032 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚336397078296576 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚333363171610112 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 _runtime‚8822.165367914œœšš2ä(- _wandbruntime‚8822  _runtime‚88228à÷&å(ñM L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16150911597568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15874818917376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 trainer/global_step‚11249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148508880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63832088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _runtime‚8822.839887036 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148508880 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚317516828192768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 376090849 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16151117974528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚333391834126336 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚302046802354176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150523921 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 227581971 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15829060565434 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 148508878 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 227581971 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 227602035 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚352576902842368 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376110915 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86692331 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚317875845880250 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788404198202394e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15874987930624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚302046785314816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15829225687046 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 376110915 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 227602035 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚336425991244800 _step‚925 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚317516828192768 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15874818917376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚498 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚317516846195712 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150524421 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86692333 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚336426015184896 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚317876028041222 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969Jyzü&æ(öM _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚336425991244800 _step‚925 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚317516828192768 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚15874818917376 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚498 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚317516846195712 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 150524421 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚86692333 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚336426015184896 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚317876028041222 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16150911597568 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚15874818917376 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 trainer/global_step‚11249 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 148508880 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚63832088 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 206376960 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 _runtime‚8822.839887036 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 148508880 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚317516828192768 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 376090849 E6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16151117974528 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚333391834126336 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚302046802354176 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 150523921 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 227581971 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚15829060565434 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 148508878 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 227581971 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 227602035 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 230317056 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚352576902842368 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 376110915 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚86692331 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚317875845880250 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 $ _timestamp‚1.788404198202394e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15874987930624 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚302046785314816 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15829225687046 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 376110915 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 227602035èíÐ2ç(- _wandbruntime‚8822  _runtime‚8822 ¥`è(Ú 0loss/FRt1/assignment_loss‚2.4663899747224955 2loss/SRqqt1/assignment_loss‚1.3582236440231403 1loss/FBt4/assignment_loss‚0.16547493346571046 /loss/FRt4/detection_loss‚1.8485384553670883 0loss/FRt2/assignment_loss‚3.7228240003952613 2loss/SRqqt3/assignment_loss‚1.5930445966751952 .loss/FBt3/detection_loss‚4.650827616453171 trainer/global_step‚11249 /loss/FBt2/detection_loss‚2.5449363365769386 0loss/FRt3/assignment_loss‚3.0930182659626007 0loss/FBt3/assignment_loss‚0.4438272434317696 1loss/SRqqt1/detection_loss‚1.6139447331428527 1loss/SRqqt2/detection_loss‚2.6298747415895813 _step‚926 2loss/SRqqt2/assignment_loss‚1.5196960088279512 1loss/FBt2/assignment_loss‚0.49967260379328593 /loss/FRt2/detection_loss‚3.1666802007418413 /loss/FRt3/detection_loss‚2.4341443008184434 1loss/SRqqt3/detection_loss‚1.7660238235666041 1loss/SRqqt4/detection_loss‚1.0846899470225686 /loss/FBt1/detection_loss‚2.8806811769803367 %loss/total_loss‚45.23424104373215 2loss/SRqqt4/assignment_loss‚1.0984393995410453 1loss/FBt1/assignment_loss‚0.10483733586166484 .loss/FRt1/detection_loss‚1.811637562341415 _runtime‚ 8822.84094945 0loss/FRt4/assignment_loss‚2.1366949054929947 epoch‚9 % _timestamp‚1.7884041982042024e+09 /loss/FBt4/detection_loss‚0.6001192369381897ÚTöeé(ß 0loss/FRt2/assignment_loss‚3.7228240003952613 2loss/SRqqt3/assignment_loss‚1.5930445966751952 .loss/FBt3/detection_loss‚4.650827616453171 trainer/global_step‚11249 /loss/FBt2/detection_loss‚2.5449363365769386 0loss/FRt3/assignment_loss‚3.0930182659626007 0loss/FBt3/assignment_loss‚0.4438272434317696 1loss/SRqqt1/detection_loss‚1.6139447331428527 1loss/SRqqt2/detection_loss‚2.6298747415895813 _step‚926 2loss/SRqqt2/assignment_loss‚1.5196960088279512 1loss/FBt2/assignment_loss‚0.49967260379328593 /loss/FRt2/detection_loss‚3.1666802007418413 /loss/FRt3/detection_loss‚2.4341443008184434 1loss/SRqqt3/detection_loss‚1.7660238235666041 1loss/SRqqt4/detection_loss‚1.0846899470225686 /loss/FBt1/detection_loss‚2.8806811769803367 %loss/total_loss‚45.23424104373215 2loss/SRqqt4/assignment_loss‚1.0984393995410453 1loss/FBt1/assignment_loss‚0.10483733586166484 .loss/FRt1/detection_loss‚1.811637562341415 _runtime‚ 8822.84094945 0loss/FRt4/assignment_loss‚2.1366949054929947 epoch‚9 % _timestamp‚1.7884041982042024e+09 /loss/FBt4/detection_loss‚0.6001192369381897 0loss/FRt1/assignment_loss‚2.4663899747224955 2loss/SRqqt1/assignment_loss‚1.3582236440231403 1loss/FBt4/assignment_loss‚0.16547493346571046 /loss/FRt4/detection_loss‚1.8485384553670883žÞ¸/{ê(‚(:’ òÃãÔŸ»¯® gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚35.0 gpu.0.powerWatts‚148.601' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.720200000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ûȃ…ë(‚(:ú òÃãÔݱï± disk.sda3.out‚ 8087.9765625 disk.sda.out‚ 8087.9765625 disk.dm-1.out‚ 8087.9765625 disk.drbd2080.in‚0% disk./.usageGB‚234.84041213989258 disk.nvme0c0n1.in‚ 416.734375 disk.rbd0.out‚ 1.0986328125 disk.sda3.in‚ 0.06640625 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.dm-0.in‚ 416.734375 disk.nvme0n1.in‚ 416.734375 disk.sda1.out‚0 disk.drbd1716.out‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0) proc.memory.percent‚16.44587516784668 network.sent‚ 2.8069297e+07$ disk.nvme0n1.out‚41235.966796875 disk.sda.in‚ 0.06640625 disk.sda2.in‚0 disk.drbd2080.out‚0! disk.dm-0.out‚41235.966796875' proc.memory.availableMB‚ 393.6484375 cpu‚25.37016862677605 network.recv‚ 3.139349e+06 disk.sda1.in‚0 disk.rbd1.in‚0& disk.nvme0c0n1.out‚41235.966796875 disk.sda2.out‚0 disk.dm-1.in‚ 0.06640625) disk./.usagePercent‚15.76033710293846 disk.drbd2265.in‚0 disk.rbd0.in‚ 0.0546875 proc.cpu.threads‚17 disk.rbd1.out‚ 23.9609375$ memory_percent‚97.59736061096191" proc.memory.rssMB‚ 2694.4921875A·#’ì(‚(:‡ ÄãÔÊ®¬® gpu.0.gpu‚41.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚132.667' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.5334 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0:Â" ]í(‚(:ü ÄãÔµÁõ± disk.sda2.out‚0 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.sda3.in‚ 0.06640625 disk.nvme0n1.in‚ 416.734375 disk.sda.in‚ 0.06640625 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.dm-0.in‚ 416.734375 disk.rbd1.in‚0 disk.sda.out‚ 8088.4453125 disk.drbd2266.in‚0! disk.dm-0.out‚41236.634765625 disk.sda1.in‚0 disk.rbd1.out‚ 24.0546875% disk./.usageGB‚234.84039306640625 disk.dm-1.in‚ 0.06640625 disk.dm-1.out‚ 8088.4453125 disk.drbd2265.in‚0 disk.rbd0.out‚ 1.0986328125 disk.nvme0c0n1.in‚ 416.734375 network.sent‚ 2.8177692e+07 disk.drbd1716.in‚0& disk.nvme0c0n1.out‚41236.634765625$ disk.nvme0n1.out‚41236.634765625 network.recv‚ 3.145495e+06 disk.rbd0.in‚ 0.0546875 disk.sda1.out‚0' proc.memory.availableMB‚ 393.6484375 proc.cpu.threads‚17* disk./.usagePercent‚15.760335822900787 disk.drbd2080.out‚0 disk.drbd1716.out‚0 disk.sda3.out‚ 8088.4453125$ memory_percent‚97.59736061096191" proc.memory.rssMB‚ 2694.4921875) proc.memory.percent‚16.ÚR¨*44587516784668 cpu‚25.369599054132387ê›xƒ2î(- _wandbruntime‚8857  _runtime‚8857ƒÙ1Êï(Ä ' lr-AdamW/pg1‚2.7911171335179798e-06 ' lr-AdamW/pg2‚2.7911171335179798e-06 trainer/global_step‚11299 % _timestamp‚1.7884042333075364e+09 _runtime‚8857.944411363 _step‚927þøÈÏð(É ' lr-AdamW/pg1‚2.7911171335179798e-06 ' lr-AdamW/pg2‚2.7911171335179798e-06 trainer/global_step‚11299 % _timestamp‚1.7884042333075364e+09 _runtime‚8857.944411363 _step‚927ŸrëJ 2ñ(- _wandbruntime‚8857  _runtime‚8857 ñZ“î'ò(èO E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87053804 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64094839 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87053806 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 228574924 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚303318018288640 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 _runtime‚8857.945351767 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚334796311169536 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚15943266101760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚334796124166656 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15943097101824 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16220344889856 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16220555474432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151148645 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64094340 % _timestamp‚1.7884042333082638e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149129169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151148144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚354063305122304 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 228554886 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚319215144468578 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149129171 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149129171 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15897143219298 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚318853045067776 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149129169 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚501 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚319215326629342 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 377684055 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚334796124166656 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15943266101760 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚354063070597632 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15943097101824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚334796311169536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 228574924 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚318853027064832 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚499 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚337842749647872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚318853045067776 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 _step‚928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚318853027064832 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 trainer/global_step‚11299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚337842725707776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 228554886 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚303318001249280 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15897308340702 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚334796124166656 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚15943097101824 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16220344889856 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16220555474432 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 151148645 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64094340 % _timestamp‚1.7884042333082638e+09 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 149129169 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 151148144 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚354063305122304 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 228554886 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚319215144468578 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149129171 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149129171 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15897143219298 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚318853045067776 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 149129169 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚501 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚319215326629342 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 377684055 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚334796124166656 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚15943266101760 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚354063070597632 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚15943097101824 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚334796311169536 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 228574924 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚318853027064832 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚499 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚337842749647872 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚318853045067776 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 _step‚928 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚318853027064832 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 trainer/global_step‚11299 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚337842725707776 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 BDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 228554886 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚303318001249280 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚15897308340702 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87053806 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 228574924 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚303318018288640 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 ^Á“2ô(- _wandbruntime‚8858  _runtime‚8858fºø&õ(òM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64100193 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 218959872 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚319242476815458 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 149141828 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚15944659253248 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚337871662596096 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚337871638656000 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚334824973685760 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151161376 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 _step‚929 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 149141830 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 228574673 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 377716501 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 377736567 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884042339823334e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15944659253248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64099688 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 228594737 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚334824973685760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151160869 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚318880296429568 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 228594737 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚507 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚354093636255744 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149141828 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87061181 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15898697505454 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 trainer/global_step‚11299 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚334824973685760 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151160869 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚318880296429568 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 228594737 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚507 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚354093636255744 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149141828 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87061181 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15898697505454 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 trainer/global_step‚11299 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 377716501 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 377736567 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 % _timestamp‚1.7884042339823334e+09 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚15944659253248 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64099688 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 P6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0¡*hXÙ2÷(- _wandbruntime‚8858  _runtime‚8858æ¼S[ø(Õ 2loss/SRqqt1/assignment_loss‚1.2861745998328715 0loss/FBt2/assignment_loss‚0.6707677281546321 0loss/FBt3/assignment_loss‚0.5659194063717595 /loss/FBt1/detection_loss‚2.5477647185325623 /loss/FBt2/detection_loss‚2.3514695926146074 %loss/total_loss‚43.62621833312322 2loss/SRqqt3/assignment_loss‚1.5611922258341855 1loss/SRqqt4/assignment_loss‚1.059381001839234 .loss/FBt4/detection_loss‚0.720560609921813 trainer/global_step‚11299 _runtime‚8858.620674195 .loss/FRt3/detection_loss‚2.455814072915486 1loss/SRqqt2/detection_loss‚2.5627736054799137 /loss/SRqqt3/detection_loss‚1.93285052054985 % _timestamp‚1.7884042339839296e+09 1loss/FBt4/assignment_loss‚0.14233551212373005 /loss/FRt1/detection_loss‚2.0840783199226416 .loss/FRt4/detection_loss‚2.265477024421499 1loss/FBt1/assignment_loss‚0.23930453184422934 .loss/FRt2/detection_loss‚3.080591283462666 1loss/SRqqt1/detection_loss‚1.7192334719655806 1loss/SRqqt4/detection_loss‚1.0851692194119096 _step‚930 0loss/FRt1/assignment_loss‚2.7805601402565285 0loss/FRt4/assignment_loss‚1.6766916658427264 2loss/SRqqt2/assignment_loss‚1.8958496323184055 /loss/FBt3/detection_loss‚2.6577690108255907 epoch‚9 0loss/FRt2/assignment_loss‚3.2415892548031278 0loss/FRt3/assignment_loss‚3.0429011838776723è¢2`ù(Ú 0loss/FRt2/assignment_loss‚3.2415892548031278 0loss/FRt3/assignment_loss‚3.0429011838776723 2loss/SRqqt1/assignment_loss‚1.2861745998328715 0loss/FBt2/assignment_loss‚0.6707677281546321 0loss/FBt3/assignment_loss‚0.5659194063717595 /loss/FBt1/detection_loss‚2.5477647185325623 /loss/FBt2/detection_loss‚2.3514695926146074 %loss/total_loss‚43.62621833312322 2loss/SRqqt3/assignment_loss‚1.5611922258341855 1loss/SRqqt4/assignment_loss‚1.059381001839234 .loss/FBt4/detection_loss‚0.720560609921813 trainer/global_step‚11299 _runtime‚8858.620674195 .loss/FRt3/detection_loss‚2.455814072915486 1loss/SRqqt2/detection_loss‚2.5627736054799137 /loss/SRqqt3/detection_loss‚1.93285052054985 % _timestamp‚1.7884042339839296e+09 1loss/FBt4/assignment_loss‚0.14233551212373005 /loss/FRt1/detection_loss‚2.0840783199226416 .loss/FRt4/detection_loss‚2.265477024421499 1loss/FBt1/assignment_loss‚0.23930453184422934 .loss/FRt2/detection_loss‚3.080591283462666 1loss/SRqqt1/detection_loss‚1.7192334719655806 1loss/SRqqt4/detection_loss‚1.0851692194119096 _step‚930 0loss/FRt1/assignment_loss‚2.7805601402565285 0loss/FRt4/assignment_loss‚1.6766916658427264 2loss/SRqqt2/assignment_loss‚1.8958496323184055 /loss/FBt3/detection_loss‚2.6577690108255907 epoch‚9¢kú(‚(:’ ÄãÔúà® gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚35.0 gpu.0.powerWatts‚150.734' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚30.146800000000002 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Žñ݈û(‚(:ý ÄãÔÜðÆ° disk.dm-1.out‚ 8089.07421875 disk.drbd2265.in‚0% disk./.usageGB‚234.84028244018555 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 disk.sda3.in‚ 0.06640625 disk.sda3.out‚ 8089.07421875 disk.rbd1.in‚0 disk.nvme0c0n1.in‚ 416.734375% disk.nvme0n1.out‚41237.0771484375 disk.dm-1.in‚ 0.06640625" proc.memory.rssMB‚ 2694.4921875 disk.sda.out‚ 8089.07421875 disk.drbd1716.out‚0$ memory_percent‚97.59979248046875 proc.cpu.threads‚17 disk.rbd0.out‚ 1.0986328125 disk.dm-0.in‚ 416.734375 network.sent‚ 2.8183861e+07* disk./.usagePercent‚15.760328398682294' disk.nvme0c0n1.out‚41237.0771484375 disk.sda2.in‚0 disk.rbd0.in‚ 0.0546875" proc.memory.availableMB‚393.25 network.recv‚ 3.149638e+06 disk.rbd1.out‚ 24.1171875 disk.sda2.out‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0) proc.memory.percent‚16.44587516784668 cpu‚25.369067890481727 disk.sda.in‚ 0.06640625 disk.drbd2266.out‚0" disk.dm-0.out‚41237.0771484375 disk.nvme0n1.in‚ 416.734375 disk.drbd1716.in‚0%|÷’ü(‚(:‡ ŸÄãÔÁÛ®¯ gpu.0.gpu‚12.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚119.733' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.9466 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Ìâ‚Ùsý(‚(:è ŸÄãÔñÀ‡± proc.cpu.threads‚17 disk.drbd2266.out‚0 network.recv‚ 3.156583e+06 disk.rbd1.in‚0 disk.nvme0c0n1.out‚ 41238.375 disk.nvme0n1.in‚ 416.734375$ memory_percent‚97.60251045227051 disk.rbd1.out‚ 24.1171875 disk.sda.out‚ 8089.5546875 disk.dm-0.in‚ 416.734375" proc.memory.rssMB‚ 2694.4921875 disk.sda1.in‚0 disk.nvme0n1.out‚ 41238.375 disk.drbd2080.in‚0 network.sent‚ 2.829263e+07 disk.sda1.out‚0 disk.sda3.out‚ 8089.5546875 disk.nvme0c0n1.in‚ 416.734375 disk.drbd1716.out‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2080.out‚0 disk.dm-1.out‚ 8089.5546875 disk.sda3.in‚ 0.06640625 disk.sda.in‚ 0.06640625 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-0.out‚ 41238.375* disk./.usagePercent‚15.760330958757637$ disk./.usageGB‚234.8403205871582 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚ 0.0546875 disk.rbd0.out‚ 1.0986328125' proc.memory.availableMB‚ 392.8046875) proc.memory.percent‚16.44587516784668 cpu‚25.368499353439155 disk.drbd2265.in‚0*ùÉ2þ(- _wandbruntime‚8893  _runtime‚8893 ;Êÿ(Ä trainer/global_step‚11349 % _timestamp‚1.7884042686999993e+09 _runtime‚8893.336864804 _step‚931 ' lr-AdamW/pg1‚2.4507149342020414e-06 ' lr-AdamW/pg2‚2.4507149342020414e-06‹$åÛÏ€)É ' lr-AdamW/pg1‚2.4507149342020414e-06 ' lr-AdamW/pg2‚2.4507149342020414e-06 trainer/global_step‚11349 % _timestamp‚1.7884042686999993e+09 _runtime‚8893.336864804 _step‚931£Þs22)- _wandbruntime‚8893  _runtime‚8893®tŸºí'‚)çO M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 229547588 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 229547588 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16291390795264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚932 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚336229263726080 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788404268700737e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149762121 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚336229450728960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87422654 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16012768424448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16291180210688 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚320216495301632 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 229567626 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87422656 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚339288397059072 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚304615177405440 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16012937424384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 379329747 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149762121 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚336229450728960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 229567626 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16012768424448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚320581775403786 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16012937424384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚339288373118976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚8893.337762037 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64365460 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚355579787854336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚355579553329664¦j](ƒ)ìO LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚336229263726080 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚541 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚320581957564550 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚304615160366080 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚15966615037706 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788404268700737e+09 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 149762121 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚336229450728960 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87422654 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16012768424448 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16291180210688 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚320216495301632 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 229567626 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87422656 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚339288397059072 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚304615177405440 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16012937424384 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 379329747 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 149762121 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚336229450728960 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 229567626 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16012768424448 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚539 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚320581775403786 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16012937424384 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚339288373118976 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _runtime‚8893.337762037 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64365460 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚355579787854336 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚355579553329664 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 229547588 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 229547588 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocatfÆ“Ê ed‚16291390795264 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 _step‚932 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚336229263726080 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16292806759424 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚304641120587776 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚320243764666368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87430033 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _runtime‚8894.003698729 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 229567375 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 379362219 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149774780 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚336257926228992 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 229567375 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚320243782669312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚355609882255360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15968169323862 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149774778 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151800768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚336258113245184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 _step‚933 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64370735 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151800270 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚320243782669312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16292596188160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16014161562624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87430031 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 379362219 trainer/global_step‚11349 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 229587439 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚320609289911638 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217ôÙ9ü&†)öM MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16292596188160 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16014161562624 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚87430031 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 379362219 trainer/global_step‚11349 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 229587439 P8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚320609289911638 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚320243764666368 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 379342153 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚498 $ _timestamp‚1.788404269366354e+09 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚339317310007296 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16292806759424 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚304641120587776 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚320243764666368 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87430033 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 _runtime‚8894.003698729 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 229567375 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 379362219 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 149774780 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚336257926228992 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 229567375 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 FDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚320243782669312 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚355609882255360 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚15968169323862 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 149774778 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 151800768 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚336258113245184 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 _step‚933 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64370735 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 151800270 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚320243782669312 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401¥® Ø2‡)- _wandbruntime‚8894  _runtime‚8894y-å]ˆ)× 2loss/SRqqt3/assignment_loss‚1.4351441444024782 /loss/FBt2/assignment_loss‚0.503090115432759 0loss/SRqqt2/detection_loss‚2.431532899246496 /loss/FBt3/detection_loss‚2.5483801901340484 /loss/FBt4/detection_loss‚0.6113537979476592 _step‚934 1loss/SRqqt2/assignment_loss‚2.002987350720693 1loss/SRqqt4/assignment_loss‚0.584558750369719 1loss/SRqqt1/detection_loss‚1.6717884607137519 epoch‚9 0loss/FRt2/assignment_loss‚3.9124454304024026 0loss/FBt3/assignment_loss‚0.3182318752631545 /loss/FRt4/detection_loss‚2.2589710280299187 /loss/FBt1/detection_loss‚3.1051537916064262 trainer/global_step‚11349 /loss/FRt3/assignment_loss‚2.677619439940299 /loss/FRt2/detection_loss‚3.1301802109788963 &loss/total_loss‚45.021979341010805 _runtime‚8894.004741526 0loss/FRt1/assignment_loss‚2.8802051599378937 2loss/SRqqt1/assignment_loss‚1.3696508532033322 /loss/FRt1/detection_loss‚2.1989787408599146 0loss/FRt4/assignment_loss‚2.0495225548744203 1loss/FBt4/assignment_loss‚0.20584839622338824 /loss/FRt3/detection_loss‚2.7065920772091037 1loss/SRqqt4/detection_loss‚1.4381916891369555 0loss/FBt1/assignment_loss‚0.3068220325388964 1loss/SRqqt3/detection_loss‚1.6904547127810392 /loss/FBt2/detection_loss‚2.9842756390571594 % _timestamp‚1.7884042693680193e+09b‰)Ü 0loss/FBt1/assignment_loss‚0.3068220325388964 1loss/SRqqt3/detection_loss‚1.6904547127810392 /loss/FBt2/detection_loss‚2.9842756390571594 % _timestamp‚1.7884042693680193e+09 2loss/SRqqt3/assignment_loss‚1.4351441444024782 /loss/FBt2/assignment_loss‚0.503090115432759 0loss/SRqqt2/detection_loss‚2.431532899246496 /loss/FBt3/detection_loss‚2.5483801901340484 /loss/FBt4/detection_loss‚0.6113537979476592 _step‚934 1loss/SRqqt2/assignment_loss‚2.002987350720693 1loss/SRqqt4/assignment_loss‚0.584558750369719 1loss/SRqqt1/detection_loss‚1.6717884607137519 epoch‚9 0loss/FRt2/assignment_loss‚3.9124454304024026 0loss/FBt3/assignment_loss‚0.3182318752631545 /loss/FRt4/detection_loss‚2.2589710280299187 /loss/FBt1/detection_loss‚3.1051537916064262 trainer/global_step‚11349 /loss/FRt3/assignment_loss‚2.677619439940299 /loss/FRt2/detection_loss‚3.1301802109788963 &loss/total_loss‚45.021979341010805 _runtime‚8894.004741526 0loss/FRt1/assignment_loss‚2.8802051599378937 2loss/SRqqt1/assignment_loss‚1.3696508532033322 /loss/FRt1/detection_loss‚2.1989787408599146 0loss/FRt4/assignment_loss‚2.0495225548744203 1loss/FBt4/assignment_loss‚0.20584839622338824 /loss/FRt3/detection_loss‚2.7065920772091037 1loss/SRqqt4/detection_loss‚1.4381916891369555¦üv[XŠ)‚(:’ ®ÄãÔ…骮 gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚142.402' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.480399999999996 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¦xd'Š‹)‚(:ÿ ®ÄãÔû‹ä° disk.rbd1.in‚0% disk.nvme0c0n1.out‚41238.66796875 disk.sda.in‚ 0.06640625 disk.drbd2265.out‚0 disk.sda1.out‚0 network.recv‚ 3.159543e+06 disk.sda.out‚ 8090.01953125 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2694.4921875 cpu‚25.367965426479394% disk./.usageGB‚234.84028244018555 disk.rbd1.out‚ 24.1494140625 disk.dm-1.in‚ 0.06640625* disk./.usagePercent‚15.760328398682294 disk.drbd2266.in‚0 disk.nvme0c0n1.in‚ 416.734375 disk.drbd1716.in‚0 disk.sda3.in‚ 0.06640625) proc.memory.percent‚16.44587516784668 proc.cpu.threads‚17 network.sent‚ 2.8297998e+07 disk.drbd1716.out‚0 disk.dm-1.out‚ 8090.01953125 disk.dm-0.out‚41238.66796875 disk.sda1.in‚0 disk.sda2.in‚0 disk.sda2.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚ 0.0546875 disk.dm-0.in‚ 416.734375 disk.nvme0n1.in‚ 416.734375# disk.nvme0n1.out‚41238.66796875 disk.drbd2265.in‚0 disk.rbd0.out‚ 1.0986328125 disk.sda3.out‚ 8090.01953125# memory_percent‚97.5996732711792( proc.memory.availableMB‚ 393.26953125©\’C’Œ)‚(:‡ ½ÄãÔ²§® gpu.0.gpu‚30.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚113.666' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.7332 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0<4±Î‹)‚(:€  ½ÄãÔ¹•ó°* disk./.usagePercent‚15.760330190735033 disk.dm-0.in‚ 416.734375 disk.rbd1.out‚ 24.2431640625 disk.dm-1.out‚ 8090.5078125 disk.drbd2266.in‚0 disk.sda3.in‚ 0.06640625' disk.nvme0c0n1.out‚41239.6044921875 disk.nvme0n1.in‚ 416.734375 disk.rbd0.out‚ 1.0986328125" disk.dm-0.out‚41239.6044921875 disk.sda3.out‚ 8090.5078125% disk.nvme0n1.out‚41239.6044921875 disk.sda.out‚ 8090.5078125 disk.rbd1.in‚0 disk.sda.in‚ 0.06640625 disk.sda2.out‚0 disk.drbd2266.out‚0 disk.drbd2265.out‚0" proc.memory.rssMB‚ 2694.4921875) proc.memory.percent‚16.44587516784668 network.sent‚ 2.840571e+07$ disk./.usageGB‚234.8403091430664 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 network.recv‚ 3.16565e+06 disk.sda1.in‚0 disk.sda2.in‚0( proc.memory.availableMB‚ 393.26171875 proc.cpu.threads‚17 disk.nvme0c0n1.in‚ 416.734375 disk.drbd1716.in‚0$ memory_percent‚97.59972095489502 cpu‚25.367430893444293 disk.rbd0.in‚ 0.0546875 disk.sda1.out‚0 disk.dm-1.in‚ 0.06640625ðnA’Ž)‚(:‡ ÌÄãÔ›™¬® gpu.0.gpu‚24.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚112.708' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚22.5416 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0D=Hr)‚(:…  ÌÄãÔ…¡„² disk.drbd2266.in‚0 disk.sda3.out‚ 8091.12890625 network.recv‚ 3.169573e+06 disk.dm-0.in‚ 416.734375 disk.sda1.in‚0 disk.rbd1.out‚ 24.3056640625" proc.memory.rssMB‚ 2694.4921875* disk./.usagePercent‚15.760335054878185% disk.nvme0n1.out‚41239.9892578125 disk.dm-1.in‚ 0.06640625 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.nvme0c0n1.in‚ 416.734375( proc.memory.availableMB‚ 393.25390625 disk.nvme0n1.in‚ 416.734375 disk.rbd0.out‚ 1.0986328125 disk.drbd1716.out‚0 disk.sda1.out‚0 disk.rbd1.in‚0 proc.cpu.threads‚17 network.sent‚ 2.841163e+07% disk./.usageGB‚234.84038162231445 disk.sda2.out‚0 disk.rbd0.in‚ 0.0546875" disk.dm-0.out‚41239.9892578125' disk.nvme0c0n1.out‚41239.9892578125 disk.drbd1716.in‚0 disk.dm-1.out‚ 8091.12890625 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.sda3.in‚ 0.06640625 disk.sda.in‚ 0.06640625 disk.sda.out‚ 8091.12890625 disk.sda2.in‚0$ memory_percent‚97.59976863861084) proc.memory.percent‚16.44587516784668 cpu‚25.3668922494228229•Í92)- _wandbruntime‚8929  _runtime‚8929B֪ʑ)Ä ' lr-AdamW/pg1‚2.1322034535935205e-06 ' lr-AdamW/pg2‚2.1322034535935205e-06 trainer/global_step‚11399 % _timestamp‚1.7884043043878055e+09 _runtime‚8929.024697812 _step‚935EÂåÏ’)É % _timestamp‚1.7884043043878055e+09 _runtime‚8929.024697812 _step‚935 ' lr-AdamW/pg1‚2.1322034535935205e-06 ' lr-AdamW/pg2‚2.1322034535935205e-06 trainer/global_step‚11399§MâTs2“)- _wandbruntime‚8929  _runtime‚8929ÖkÔ?”)èO SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150395069 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚340734044470272 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚305912319482880 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 230560328 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 152426659 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87791506 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16036251977518 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 % _timestamp‚1.7884043043885179e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 R>DeviceStatsMonitor.on_train__’d‚¯batch_start/active_bytes.all.freed‚337662403285504 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚357096329904640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚321579963538432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 230540290 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚321948588499758 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 trainer/global_step‚11399 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚337662590288384 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚321579981541376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚321579963538432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚337662403285504 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 230540290 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150395071 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150395069 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚357096091185664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 380955399 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16036086856114 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚514 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64634641 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16362070655488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚337662590288384 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚321948406338994 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16082439747072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87791504 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 380935359 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚321579981541376 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152426145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚936 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16082439747072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16362285434368 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16082608747008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0Šƒ‹ó'•)íO MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚340734044470272 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚305912319482880 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 230560328 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 152426659 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚87791506 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16036251977518 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 % _timestamp‚1.7884043043885179e+09 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚337662403285504 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚357096329904640 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 214778880 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚321579963538432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 230540290 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚321948588499758 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 trainer/global_step‚11399 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚337662590288384 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚321579981541376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚321579963538432 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚337662403285504 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 230540290 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 150395071 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 150395069 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚357096091185664 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 380955399 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16036086856114 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚514 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64634641 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16362070655488 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚337662590288384 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚321948406338994 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16082439747072 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚87791504 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚512 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 380935359 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚321579981541376 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 152426145 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _step‚936 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16082439747072 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16362285434368 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16082608747008 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 150395069¨Ñ{ÿ¬2–)- _wandbruntime‚8929  _runtime‚8929Á£„ø&—)òM B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 150407730 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 150407728 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 150407728 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 152439673 _runtime‚8929.694547477 _step‚937 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚321607232903168 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚337691065788416 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚321975738685874 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152439190 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚337691252804608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16363713646592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚483 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚321975920846846 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 230560077 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16084001898496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340762933478400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 trainer/global_step‚11399 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚481 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224025600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 380967805 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 230580141 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚321607250906112 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚357126671065088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚321607250906112 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87798883 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64640790 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚321607232903168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64640309 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16037476020658 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200085504 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚305938279704576 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 380967805 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 380987871 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16084001898496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚357126447039488 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 380987871 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884043050572405e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚305938262665216 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150407730 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16083832885248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16083832885248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 P,é˜)÷M >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 380987871 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 % _timestamp‚1.7884043050572405e+09 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚305938262665216 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 150407730 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16083832885248 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16083832885248 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 PDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_Zôy0ìpool.peak‚ 410712064 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚321975738685874 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 152439190 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚337691252804608 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16363713646592 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚483 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚321975920846846 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 230560077 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16084001898496 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚340762933478400 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 trainer/global_step‚11399 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚481 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 224025600 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 380967805 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 230580141 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚321607250906112 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚357126671065088 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚321607250906112 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚87798883 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚64640790 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚321607232903168 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64640309 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16037476020658 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 200085504 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚305938279704576 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 380967805 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 380987871 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16084001898496 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚357126447039488©4×r2™)- _wandbruntime‚8929  _runtime‚8929+nMã\š)Ö 0loss/FBt2/assignment_loss‚0.2729588356791055 1loss/SRqqt2/detection_loss‚2.3919456467546265 .loss/FRt2/assignment_loss‚3.72177016293561 /loss/FBt1/detection_loss‚2.2215860394331126 /loss/FBt3/detection_loss‚2.7063872483041553 0loss/FBt4/assignment_loss‚0.2034528210488408 .loss/FRt2/detection_loss‚3.002978159321679 2loss/SRqqt3/assignment_loss‚1.3699481681637142 .loss/FBt3/assignment_loss‚0.38026835081271 /loss/FRt4/detection_loss‚1.6621491989873824 1loss/SRqqt3/detection_loss‚1.8550847153300825 _runtime‚8929.695691575 _step‚938 1loss/SRqqt1/detection_loss‚1.8281761235660976 /loss/FBt4/detection_loss‚0.6976574604110471 epoch‚9 0loss/FRt3/assignment_loss‚3.1248175929421964 0loss/FRt4/assignment_loss‚1.4995007514953613 2loss/SRqqt1/assignment_loss‚1.5125011740459335 1loss/SRqqt4/detection_loss‚0.8824757609358975 2loss/SRqqt2/assignment_loss‚1.8621647820390503 2loss/SRqqt4/assignment_loss‚1.3496181519938186 /loss/FRt1/detection_loss‚2.2457755414095333 % _timestamp‚1.7884043050589674e+09 /loss/FRt3/detection_loss‚2.4959149840085404 &loss/total_loss‚43.248572976047484 /loss/FBt2/detection_loss‚2.7339917182922364 trainer/global_step‚11399 /loss/FRt1/assignment_loss‚3.011005008474309 1loss/FBt1/assignment_loss‚0.21644457966244074p>Uüa›)Û % _timestamp‚1.7884043050589674e+09 /loss/FRt3/detection_loss‚2.4959149840085404 &loss/total_loss‚43.248572976047484 /loss/FBt2/detection_loss‚2.7339917182922364 trainer/global_step‚11399 /loss/FRt1/assignment_loss‚3.011005008474309 1loss/FBt1/assignment_loss‚0.21644457966244074 0loss/FBt2/assignment_loss‚0.2729588356791055 1loss/SRqqt2/detection_loss‚2.3919456467546265 .loss/FRt2/assignment_loss‚3.72177016293561 /loss/FBt1/detection_loss‚2.2215860394331126 /loss/FBt3/detection_loss‚2.7063872483041553 0loss/FBt4/assignment_loss‚0.2034528210488408 .loss/FRt2/detection_loss‚3.002978159321679 2loss/SRqqt3/assignment_loss‚1.3699481681637142 .loss/FBt3/assignment_loss‚0.38026835081271 /loss/FRt4/detection_loss‚1.6621491989873824 1loss/SRqqt3/detection_loss‚1.8550847153300825 _runtime‚8929.695691575 _step‚938 1loss/SRqqt1/detection_loss‚1.8281761235660976 /loss/FBt4/detection_loss‚0.6976574604110471 epoch‚9 0loss/FRt3/assignment_loss‚3.1248175929421964 0loss/FRt4/assignment_loss‚1.4995007514953613 2loss/SRqqt1/assignment_loss‚1.5125011740459335 1loss/SRqqt4/detection_loss‚0.8824757609358975 2loss/SRqqt2/assignment_loss‚1.8621647820390503 2loss/SRqqt4/assignment_loss‚1.3496181519938186 /loss/FRt1/detection_loss‚2.2457755414095333ª@htâœ)‚(:’ ÛÄãÔÝ”® gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚134.552' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.910400000000003 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÃÄò ’)‚(:‡  ÛÄãÔ¸¬ð° disk.rbd0.out‚ 1.0986328125 disk.sda3.in‚ 0.06640625 disk.dm-1.out‚ 8091.52734375 disk.drbd2266.in‚0 disk.sda1.out‚0( disk.nvme0c0n1.out‚41240.81298828125 disk.drbd2265.out‚0 disk.dm-0.in‚ 416.734375" proc.memory.rssMB‚ 2694.4921875 proc.cpu.threads‚17 disk.dm-1.in‚ 0.06640625 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd1716.in‚0# disk.dm-0.out‚41240.81298828125 disk.nvme0n1.in‚ 416.734375 network.recv‚ 3.174404e+06 disk.sda.out‚ 8091.52734375 disk.sda2.in‚0 disk.drbd2266.out‚0 disk.sda3.out‚ 8091.52734375 disk.nvme0c0n1.in‚ 416.734375$ memory_percent‚97.59500026702881( proc.memory.availableMB‚ 394.03515625 network.sent‚ 2.8417949e+07 disk.drbd1716.out‚0) proc.memory.percent‚16.44587516784668$ disk./.usageGB‚234.8403778076172 disk.sda.in‚ 0.06640625 disk.sda1.in‚0 disk.rbd1.in‚0& disk.nvme0n1.out‚41240.81298828125 cpu‚25.36636878746644 disk.rbd1.out‚ 24.3056640625* disk./.usagePercent‚15.760334798870652 disk.sda2.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚ 0.0546875˜cè2ž)‚(:’ êÄãÔÀÏ® gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚121.088' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚24.217599999999997 gpu.0.smClock‚1305 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0&LœŽŸ)‚(:ƒ  êÄãÔڹ˰ disk.sda.out‚ 8094.63671875* disk./.usagePercent‚15.760328910697364 disk.dm-1.out‚ 8094.63671875 disk.drbd2266.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda.in‚ 0.06640625# proc.memory.availableMB‚394.125" proc.memory.rssMB‚ 2694.4921875 cpu‚25.365814198924713# disk.dm-0.out‚41241.33154296875 disk.rbd1.in‚0 disk.rbd1.out‚ 24.337890625( disk.nvme0c0n1.out‚41241.33154296875 disk.sda2.in‚0 disk.drbd1716.in‚0$ memory_percent‚97.59445190429688 disk.sda3.in‚ 0.06640625 disk.nvme0c0n1.in‚ 416.734375 proc.cpu.threads‚17 disk.sda1.out‚0 disk.nvme0n1.in‚ 416.734375 disk.sda2.out‚0 disk.sda1.in‚0 disk.rbd0.in‚ 0.0546875 disk.sda3.out‚ 8094.63671875& disk.nvme0n1.out‚41241.33154296875 disk.dm-1.in‚ 0.06640625) proc.memory.percent‚16.44587516784668 network.sent‚ 2.8525869e+07 disk.drbd2080.out‚0 disk.drbd1716.out‚0 network.recv‚ 3.179874e+06% disk./.usageGB‚234.84029006958008 disk.drbd2080.in‚0 disk.rbd0.out‚ 1.0986328125 disk.dm-0.in‚ 416.734375PL õ2 )- _wandbruntime‚8964  _runtime‚8964ê° Ê¡)Ä ' lr-AdamW/pg1‚1.8356490690075941e-06 ' lr-AdamW/pg2‚1.8356490690075941e-06 trainer/global_step‚11449 % _timestamp‚1.7884043398525488e+09 _runtime‚8964.489390481 _step‚939UÌÆÏ¢)É ' lr-AdamW/pg1‚1.8356490690075941e-06 ' lr-AdamW/pg2‚1.8356490690075941e-06 trainer/global_step‚11449 % _timestamp‚1.7884043398525488e+09 _runtime‚8964.489390481 _step‚939«$;¿2£)- _wandbruntime‚8964  _runtime‚8964ˆ€‚]í'¤)çO F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _step‚940 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64905500 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚322943449778176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 $ _timestamp‚1.788404339853213e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚339095729847808 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16152111069696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151028019 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚322943449778176 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153066366 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚358612644114432 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 382561011 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚358612884930560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 231553030 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64906010 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16152280069632 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16105558674522 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚323315219434966 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚342179667941376 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚339095542844928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 231553030 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16152280069632 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 231532992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 231532992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 _runtime‚8964.490303212 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151028021 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153065854 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚323315037274202 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚307209478599680 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16105723795926=ò'¥)ìO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚322943449778176 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153066366 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚358612644114432 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 382561011 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚358612884930560 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 231553030 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚64906010 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16152280069632 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16105558674522 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚323315219434966 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚342179667941376 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚339095542844928 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 231553030 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16152280069632 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 231532992 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 231532992 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 _runtime‚8964.490303212 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151028021 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153065854 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚323315037274202 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚307209478599680 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16105723795926 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 _step‚940 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚64905500 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚322943449778176 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 $ _timestamp‚1.788404339853213e+09 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚339095729847808 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16152111069696 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151028019 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160¬h©[r2¦)- _wandbruntime‚8965  _runtime‚8965éWM§)ñM I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚2+º@ª&0064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚502 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88167733 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64910858 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 231552779 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚500 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚342208580889600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚307235421782016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151040680 trainer/global_step‚11449 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚322970701139968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚323342369621082 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 382593457 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88167731 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16434605809152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 231552779 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 $ _timestamp‚1.788404340525177e+09 _runtime‚8965.162508343 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 382613523 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16153504207872 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16434391043584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16106947839066 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153078589 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚322970719142912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16153673221120 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚339124392364032 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16153673221120 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 382613523 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚358642971933184 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 _step‚941 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 231572843 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 382593457 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151040678(|»pü&¨)öM F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 _step‚941 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 231572843 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 382593457 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151040678 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚502 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚88167733 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 EDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚64910858 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 231552779 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚500 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚342208580889600 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚307235421782016 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151040680 trainer/global_step‚11449 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚322970701139968 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚323342369621082 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 382593457 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88167731 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16434605809152 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 FDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 231552779 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 $ _timestamp‚1.788404340525177e+09 _runtime‚8965.162508343 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 382613523 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16153504207872 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16434391043584 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16106947839066 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 BDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153078589 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚322970719142912 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16153673221120 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚339124392364032 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16153673221120 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 382613523 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚358642971933184­ÖÉ2©)- _wandbruntime‚8965  _runtime‚8965Aˆ³\ª)Ö 0loss/FRt4/assignment_loss‚1.9128282911011152 0loss/SRqqt4/assignment_loss‚1.21314719578985 /loss/FRt4/detection_loss‚1.6463110345814909 0loss/FRt3/assignment_loss‚2.6389373734593393 0loss/FBt1/assignment_loss‚0.4764235848727237 2loss/FBt3/assignment_loss‚0.042664406821131706 _step‚942 /loss/FRt2/detection_loss‚2.8817229978740215 /loss/FRt3/detection_loss‚2.5016959995031356 /loss/FBt1/detection_loss‚2.1806677324431285 %loss/total_loss‚46.37275561736238 2loss/SRqqt1/assignment_loss‚1.5273318381083232 1loss/SRqqt1/detection_loss‚1.6604760556861207 _runtime‚ 8965.1635138 0loss/FRt1/assignment_loss‚3.9112344991486028 /loss/FRt1/detection_loss‚2.0668939472937544 1loss/SRqqt2/detection_loss‚2.5227851366996763 1loss/SRqqt3/detection_loss‚1.6989000314160396 /loss/FBt4/detection_loss‚0.6484767519351509 epoch‚9 trainer/global_step‚11449 0loss/FRt2/assignment_loss‚4.4455916583538055 0loss/FBt2/assignment_loss‚0.3053616016710293 /loss/FBt3/detection_loss‚3.5171464681625366 1loss/SRqqt2/assignment_loss‚1.808576534986496 2loss/SRqqt3/assignment_loss‚1.2972601684496592 1loss/FBt4/assignment_loss‚0.16011793576206682 0loss/SRqqt4/detection_loss‚1.000542112710801 .loss/FBt2/detection_loss‚4.307662260532379 % _timestamp‚1.7884043405267873e+09ݧXœa«)Û 1loss/SRqqt1/detection_loss‚1.6604760556861207 _runtime‚ 8965.1635138 0loss/FRt1/assignment_loss‚3.9112344991486028 /loss/FRt1/detection_loss‚2.0668939472937544 1loss/SRqqt2/detection_loss‚2.5227851366996763 1loss/SRqqt3/detection_loss‚1.6989000314160396 /loss/FBt4/detection_loss‚0.6484767519351509 epoch‚9 trainer/global_step‚11449 0loss/FRt2/assignment_loss‚4.4455916583538055 0loss/FBt2/assignment_loss‚0.3053616016710293 /loss/FBt3/detection_loss‚3.5171464681625366 1loss/SRqqt2/assignment_loss‚1.808576534986496 2loss/SRqqt3/assignment_loss‚1.2972601684496592 1loss/FBt4/assignment_loss‚0.16011793576206682 0loss/SRqqt4/detection_loss‚1.000542112710801 .loss/FBt2/detection_loss‚4.307662260532379 % _timestamp‚1.7884043405267873e+09 0loss/FRt4/assignment_loss‚1.9128282911011152 0loss/SRqqt4/assignment_loss‚1.21314719578985 /loss/FRt4/detection_loss‚1.6463110345814909 0loss/FRt3/assignment_loss‚2.6389373734593393 0loss/FBt1/assignment_loss‚0.4764235848727237 2loss/FBt3/assignment_loss‚0.042664406821131706 _step‚942 /loss/FRt2/detection_loss‚2.8817229978740215 /loss/FRt3/detection_loss‚2.5016959995031356 /loss/FBt1/detection_loss‚2.1806677324431285 %loss/total_loss‚46.37275561736238 2loss/SRqqt1/assignment_loss‚1.5273318381083232®ÁzK»¬)‚(:’ ùÄãÔæ­„® gpu.0.gpu‚35.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚146.709' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.341800000000003 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¹žp­)‚(:‚  ùÄãÔ¿Èá° disk.dm-1.in‚ 0.06640625 disk.drbd2080.in‚0 disk.rbd0.out‚ 1.0986328125 disk.sda.in‚ 0.06640625 disk.rbd0.in‚ 0.0546875 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.sda3.in‚ 0.06640625 disk.rbd1.in‚0 disk.nvme0n1.in‚ 416.734375& disk.nvme0n1.out‚41249.61279296875 disk.sda.out‚ 8096.296875" proc.memory.rssMB‚ 2694.4921875 disk.drbd2266.out‚0( disk.nvme0c0n1.out‚41249.61279296875( proc.memory.availableMB‚ 393.81640625 cpu‚25.365286553497096 proc.cpu.threads‚17* disk./.usagePercent‚15.760393680603526 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 8096.296875 disk.rbd1.out‚ 24.517578125 disk.sda2.out‚0) proc.memory.percent‚16.44587516784668 network.sent‚ 2.8532073e+07 network.recv‚ 3.184568e+06 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.dm-0.in‚ 416.734375# disk.dm-0.out‚41249.61279296875$ memory_percent‚97.59633541107178 disk.sda1.out‚0 disk.sda2.in‚0% disk./.usageGB‚234.84125518798828 disk.dm-1.out‚ 8096.296875 disk.nvme0c0n1.in‚ 416.734375·Në’®)‚(:‡ ˆÅãÔ‘‰•® gpu.0.gpu‚31.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚35.0 gpu.0.powerWatts‚144.824' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.9648 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0âÕ¸ÛŒ¯)‚(:  ˆÅãÔ˵„² disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚ 0.0546875! disk.dm-0.out‚41249.970703125 disk.nvme0c0n1.in‚ 416.734375 disk.sda.out‚ 8097.01953125( proc.memory.availableMB‚ 393.62890625 cpu‚25.36478195312079 network.recv‚ 3.190548e+06 disk.sda2.in‚0 disk.dm-1.out‚ 8097.01953125 disk.sda1.out‚0 disk.sda3.out‚ 8097.01953125 disk.rbd1.in‚0$ memory_percent‚97.59747982025146 network.sent‚ 2.8640446e+07& disk.nvme0c0n1.out‚41249.970703125$ disk.nvme0n1.out‚41249.970703125 disk.sda.in‚ 0.06640625% disk./.usageGB‚234.84134674072266 disk.drbd2265.in‚0 disk.nvme0n1.in‚ 416.734375 proc.cpu.threads‚17 disk.sda1.in‚0 disk.rbd1.out‚ 24.580078125 disk.sda2.out‚0 disk.dm-1.in‚ 0.06640625 disk.dm-0.in‚ 416.734375* disk./.usagePercent‚15.760399824784347 disk.drbd2266.out‚0 disk.drbd2265.out‚0" proc.memory.rssMB‚ 2694.4921875) proc.memory.percent‚16.44587516784668 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚ 1.0986328125 disk.sda3.in‚ 0.06640625²Ÿ’°)‚(:‡ —ÅãÔâæ® gpu.0.gpu‚33.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚134.263' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚26.8526 gpu.0.smClock‚1350 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0«ø²ßŠ±)‚(:ÿ —ÅãÔŒš‹±' disk.nvme0c0n1.out‚41250.5634765625" proc.memory.rssMB‚ 2694.4921875 disk.drbd1716.in‚0 disk.drbd1716.out‚0 disk.rbd0.in‚ 0.0546875 network.recv‚ 3.195065e+06 disk.dm-1.out‚ 8097.41796875% disk.nvme0n1.out‚41250.5634765625 disk.drbd2080.in‚0# proc.memory.availableMB‚393.875 disk.nvme0n1.in‚ 416.734375 disk.drbd2265.in‚0 disk.dm-0.in‚ 416.734375" disk.dm-0.out‚41250.5634765625 disk.sda1.out‚0 disk.sda3.out‚ 8097.41796875 disk.rbd1.out‚ 24.580078125) proc.memory.percent‚16.44587516784668% disk./.usageGB‚234.84132766723633 disk.sda2.in‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2266.in‚0 disk.rbd1.in‚0$ memory_percent‚97.59597778320312 proc.cpu.threads‚17 disk.sda.out‚ 8097.41796875 disk.drbd2266.out‚0 disk.drbd2080.out‚0 cpu‚25.36423655774742 disk.sda.in‚ 0.06640625 disk.sda2.out‚0 disk.sda3.in‚ 0.06640625 network.sent‚ 2.8646711e+07* disk./.usagePercent‚15.760398544746677 disk.drbd2265.out‚0 disk.rbd0.out‚ 1.0986328125 disk.sda1.in‚0 disk.nvme0c0n1.in‚ 416.734375´±åÃ2²)- _wandbruntime‚9000  _runtime‚9000œÂyÓʳ)Ä % _timestamp‚1.7884043755198016e+09 _runtime‚9000.156619185 _step‚943 ' lr-AdamW/pg1‚1.5611135819335156e-06 ' lr-AdamW/pg2‚1.5611135819335156e-06 trainer/global_step‚11499ö¯HÏ´)É ' lr-AdamW/pg1‚1.5611135819335156e-06 ' lr-AdamW/pg2‚1.5611135819335156e-06 trainer/global_step‚11499 % _timestamp‚1.7884043755198016e+09 _runtime‚9000.156619185 _step‚943¯OpÙ2µ)- _wandbruntime‚9000  _runtime‚9000#Ì´6š¶)èO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 232525694 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚340528682404352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16221951392256 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153706351 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153705807 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚308506637716480 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚324306918014976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚324681850370174 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 232545732 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚308506654755840 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚340528869407232 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16503843543040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16175030492930 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88529204 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _runtime‚9000.157497772 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 trainer/global_step‚11499 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151660971 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚343625339292672 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚3405ü¨ìT28682404352 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151660971 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16221782392320 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65177145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151660969 _step‚944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16221782392320 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 232545732 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65176603 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 384206703 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88529206 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚324306918014976 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16221951392256 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16175195614334 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 232525694 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚324306900012032 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚360129158895616ùVAtó'·)íO O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 232545732 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚308506654755840 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚340528869407232 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16503843543040 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16175030492930 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88529204 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _runtime‚9000.157497772 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 trainer/global_step‚11499 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 151660971 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚343625339292672 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚340528682404352 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 151660971 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16221782392320 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65177145 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 151660969 _step‚944 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16221782392320 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 232545732 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65176603 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 384206703 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88529206 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚324306918014976 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16221951392256 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16175195614334 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 232525694 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚324306900012032 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚360129158895616 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚542 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 232525694 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 BDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚340528682404352 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16221951392256 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 153706351 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 153705807 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚308506637716480 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚324306918014976 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚324681850370174°̈́ç2¸)- _wandbruntime‚9000  _runtime‚9000–G~æø&¹)òM F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚324709182717262 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚340557531923456 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚324334187379712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 384239175 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 384239175 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚324334169376768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚340557531923456 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16223344543744 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚340557344907264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7884043761920853e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _runtime‚9000.829480263 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 232545481 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16223344543744 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚343654228300800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16223175530496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151673630 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151673628 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88536581 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151673628 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚324334187379712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 384219109 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚324334169376768 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153718518 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚499 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16176584779086 _step‚945 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65182434 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚324709000556290 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16505470395904 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153719017 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65181937 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚360159722636800 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16223175530496 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0,[nlº)÷M I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16223175530496 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 151673630 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 151673628 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88536581 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 151673628 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚324334187379712 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 384219109 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚324334169376768 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 153718518 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚499 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16176584779086 _step‚945 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65182434 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚324709000556290 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16505470395904 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 153719017 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65181937 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚360159722636800 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16223175530496 F8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚324709182717262 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚340557531923456 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ^JDevicÝ@r‘ eStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚324334187379712 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 384239175 EDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 384239175 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚324334169376768 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚340557531923456 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16223344543744 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚340557344907264 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 % _timestamp‚1.7884043761920853e+09 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _runtime‚9000.829480263 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 232545481 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16223344543744 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚343654228300800 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0±¹óZí2»)- _wandbruntime‚9000  _runtime‚9000Òm_lZ¼)Ô /loss/FRt4/detection_loss‚1.9502065957112367 _step‚946 0loss/FRt1/assignment_loss‚2.4757786797625676 1loss/FBt4/assignment_loss‚0.19014514087352158 .loss/FRt3/detection_loss‚2.161777159118134 0loss/SRqqt1/detection_loss‚1.666456217364389 1loss/SRqqt4/detection_loss‚1.2227530047349946 /loss/FRt3/assignment_loss‚2.417298868946407 1loss/SRqqt3/detection_loss‚1.4803148807336888 %loss/total_loss‚41.90591968196827 /loss/FBt3/detection_loss‚3.0134366266429424 trainer/global_step‚11499 2loss/SRqqt1/assignment_loss‚0.9207973654324911 2loss/SRqqt2/assignment_loss‚1.7483243482077824 1loss/FBt3/assignment_loss‚0.23988989456597665 _runtime‚9000.830603011 2loss/SRqqt4/assignment_loss‚1.3327646527150776 0loss/FBt1/assignment_loss‚0.5069118993803841 1loss/SRqqt2/detection_loss‚2.3327295184135437 .loss/FBt1/detection_loss‚2.353651762008667 /loss/FBt4/detection_loss‚1.0491115549682744 % _timestamp‚1.7884043761938496e+09 -loss/FRt1/detection_loss‚1.96058314879031 0loss/FRt4/assignment_loss‚1.7564718584681667 .loss/FBt2/detection_loss‚2.932587211782282 epoch‚9 .loss/FRt2/detection_loss‚2.934095910101226 0loss/FRt2/assignment_loss‚3.6949967723904233 2loss/SRqqt3/assignment_loss‚1.2518088870371382 0loss/FBt2/assignment_loss‚0.3130277238186507ä²?É_½)Ù .loss/FRt2/detection_loss‚2.934095910101226 0loss/FRt2/assignment_loss‚3.6949967723904233 2loss/SRqqt3/assignment_loss‚1.2518088870371382 0loss/FBt2/assignment_loss‚0.3130277238186507 /loss/FRt4/detection_loss‚1.9502065957112367 _step‚946 0loss/FRt1/assignment_loss‚2.4757786797625676 1loss/FBt4/assignment_loss‚0.19014514087352158 .loss/FRt3/detection_loss‚2.161777159118134 0loss/SRqqt1/detection_loss‚1.666456217364389 1loss/SRqqt4/detection_loss‚1.2227530047349946 /loss/FRt3/assignment_loss‚2.417298868946407 1loss/SRqqt3/detection_loss‚1.4803148807336888 %loss/total_loss‚41.90591968196827 /loss/FBt3/detection_loss‚3.0134366266429424 trainer/global_step‚11499 2loss/SRqqt1/assignment_loss‚0.9207973654324911 2loss/SRqqt2/assignment_loss‚1.7483243482077824 1loss/FBt3/assignment_loss‚0.23988989456597665 _runtime‚9000.830603011 2loss/SRqqt4/assignment_loss‚1.3327646527150776 0loss/FBt1/assignment_loss‚0.5069118993803841 1loss/SRqqt2/detection_loss‚2.3327295184135437 .loss/FBt1/detection_loss‚2.353651762008667 /loss/FBt4/detection_loss‚1.0491115549682744 % _timestamp‚1.7884043761938496e+09 -loss/FRt1/detection_loss‚1.96058314879031 0loss/FRt4/assignment_loss‚1.7564718584681667 .loss/FBt2/detection_loss‚2.932587211782282 epoch‚9²Üß™¾)‚(:… ¦ÅãÔ¸¯® gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚146.42' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚29.284 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¢P’Ɉ¿)‚(:ý ¦ÅãÔ¼Éé° disk.drbd2080.in‚0 cpu‚25.363716691756796 disk.rbd1.out‚ 24.6123046875 disk.sda.in‚ 0.06640625 disk.sda2.out‚0 disk.dm-1.in‚ 0.06640625 disk.dm-1.out‚ 8098.1953125 disk.rbd0.in‚ 0.0546875 disk.dm-0.out‚41251.05078125 network.recv‚ 3.198561e+06) proc.memory.percent‚16.44587516784668 proc.cpu.threads‚17 disk.nvme0n1.in‚ 416.734375 disk.drbd2265.in‚0 disk.rbd0.out‚ 1.0986328125 disk.sda1.out‚0$ memory_percent‚97.59633541107178* disk./.usagePercent‚15.760398544746677% disk.nvme0c0n1.out‚41251.05078125 disk.sda.out‚ 8098.1953125 disk.drbd2266.out‚0 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 8098.1953125 disk.drbd2080.out‚0 disk.dm-0.in‚ 416.734375 disk.rbd1.in‚0# disk.nvme0n1.out‚41251.05078125 disk.sda2.in‚0 disk.drbd2266.in‚0 network.sent‚ 2.8652118e+07 disk.nvme0c0n1.in‚ 416.734375 disk.drbd1716.in‚0 disk.sda3.in‚ 0.06640625( proc.memory.availableMB‚ 393.81640625" proc.memory.rssMB‚ 2694.4921875% disk./.usageGB‚234.84132766723633 disk.drbd1716.out‚0`‚N?’À)‚(:‡ µÅãÔůƒ® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚138.329' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚27.6658 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÆûW¾–Á)‚(:‹  µÅãÔáÆÑ° disk.dm-1.in‚ 0.06640625 disk.sda1.in‚0( proc.memory.availableMB‚ 393.56640625 disk.drbd2265.in‚0* disk./.usagePercent‚15.760484819285715% disk./.usageGB‚234.84261322021484 disk.sda3.out‚ 8099.71484375 disk.sda2.in‚0 disk.dm-1.out‚ 8099.71484375 disk.drbd2080.out‚0 disk.rbd0.in‚ 0.0546875$ memory_percent‚97.59786128997803" proc.memory.rssMB‚ 2694.4921875 disk.nvme0n1.in‚ 416.7421875 disk.drbd1716.in‚0 disk.dm-0.in‚ 416.7421875# disk.dm-0.out‚41264.47119140625 disk.sda1.out‚0) proc.memory.percent‚16.44587516784668 proc.cpu.threads‚17 disk.rbd1.out‚ 24.7060546875& disk.nvme0n1.out‚41264.47119140625 disk.drbd2266.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚ 0.06640625 cpu‚25.363198100737545 network.sent‚ 2.8760765e+07! disk.nvme0c0n1.in‚ 416.7421875( disk.nvme0c0n1.out‚41264.47119140625 disk.sda2.out‚0 network.recv‚ 3.20462e+06 disk.rbd1.in‚0 disk.sda.in‚ 0.06640625 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚ 1.0986328125 disk.sda.out‚ 8099.71484375 disk.drbd1716.out‚0õ]ëµ2Â)- _wandbruntime‚9035  _runtime‚9035t`˜ÊÃ)Ä % _timestamp‚1.7884044110110528e+09 _runtime‚9035.647875075 _step‚947 ' lr-AdamW/pg1‚1.3086542051553078e-06 ' lr-AdamW/pg2‚1.3086542051553078e-06 trainer/global_step‚11549òS•øÏÄ)É _step‚947 ' lr-AdamW/pg1‚1.3086542051553078e-06 ' lr-AdamW/pg2‚1.3086542051553078e-06 trainer/global_step‚11549 % _timestamp‚1.7884044110110528e+09 _runtime‚9035.647875075³œ¯2Å)- _wandbruntime‚9035  _runtime‚9035Ë'Wî'Æ)èO `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚325670386251776 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16291453714944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚88898054 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚326048299144618 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚517 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚345070986703872 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚341961821963776 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚341962008966656 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 233518396 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚325670368248832 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16574865363968 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16574667362304 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 trainer/global_step‚11549 _step‚948 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 385812315 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 233538434 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65449477 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 % _timestamp‚1.7884044110117385e+09 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚515 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚309803813872640 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 152293921 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88898056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚325670386251776 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 233538434 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚326048481305382 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152293919 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚361645630126080 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚341962008966656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 233518396 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚325670368248832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198001664 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 154347016 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152293921 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚361645852067840 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16291622714880 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚345070962763776 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16291453714944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16291622714880 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221941760 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚309803796833280 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 152293921 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚88898056 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚325670386251776 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 233538434 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚326048481305382 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152293919 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚361645630126080 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚341962008966656 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 233518396 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚325670368248832 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 198001664 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 154347016 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 152293921 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚361645852067840 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16291622714880 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚345070962763776 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16291453714944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16291622714880 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 221941760 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚309803796833280 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚341961821963776 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚341962008966656 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 233518396 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚325670368248832 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16574865363968 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416´ütmç2È)- _wandbruntime‚9036  _runtime‚9036ÑÄxfÄ É)ñM ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚326075631491498 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 385864827 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16576085218816 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152306580 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚341990484466688 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 $ _timestamp‚1.788404411680161e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚345099899652096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16246056597494 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚341990671482880 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚325697655616512 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 trainer/global_step‚11549 _step‚949 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 385844761 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65454826 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 QCDeviceStaÚh3tsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 233558247 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 233538183 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚325697637613568 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 385844761 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚361675960930816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152306578 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16292846853120 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154359737 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚345099875712000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚325697655616512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16293015866368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152306578 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚325697637613568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚341990671482880 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 385864827 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 233538183 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚361676195442176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16293015866368 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚309829757054976 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16245891475882 _runtime‚9036.317424672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88905431 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217)¡SÕü&Ê)öM E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16293015866368 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚309829757054976 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16245891475882 _runtime‚9036.317424672 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚88905431 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚326075631491498 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 385864827 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16576085218816 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152306580 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚341990484466688 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 $ _timestamp‚1.788404411680161e+09 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚345099899652096 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16246056597494 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚341990671482880 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚325697655616512 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 trainer/global_step‚11549 _step‚949 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 385844761 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65454826 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 233558247 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 233538183 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚325697637613568 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 385844761 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚361675960930816 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152306578 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16292846853120 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154359737 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚345099875712000 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚325697655616512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16293015866368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 234511360 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 BDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152306578 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 210571264 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚325697637613568 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚341990671482880 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 385864827 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 233538183 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚361676195442176 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2µˆô­2Ë)- _wandbruntime‚9036  _runtime‚9036žiK\YÌ)Ó 0loss/FBt1/assignment_loss‚0.3011743876617402 .loss/FRt4/detection_loss‚1.666017865606894 -loss/FBt3/detection_loss‚3.24802266061306 &loss/total_loss‚42.594033677051044 _runtime‚9036.318391726 _step‚950 0loss/FRt2/assignment_loss‚3.7539447405759026 0loss/FRt3/assignment_loss‚2.3860227251052857 /loss/FRt1/detection_loss‚2.1167276748951465 2loss/SRqqt2/assignment_loss‚1.9749977945378332 .loss/FRt2/detection_loss‚2.873834443442962 1loss/SRqqt4/detection_loss‚0.9306231113698553 .loss/FBt1/detection_loss‚2.492834687232971 epoch‚9 /loss/FRt1/assignment_loss‚3.230871613989485 1loss/SRqqt1/assignment_loss‚1.309479199046659 2loss/SRqqt3/assignment_loss‚1.2801193727646023 2loss/SRqqt4/assignment_loss‚1.0508179530665718 0loss/FBt3/assignment_loss‚0.4787112697315479 1loss/FBt4/assignment_loss‚0.28197732746761334 1loss/SRqqt1/detection_loss‚1.9244119777928952 trainer/global_step‚11549 0loss/FRt4/assignment_loss‚1.4123392167190711 1loss/FBt2/assignment_loss‚0.29077739567040745 /loss/FBt2/detection_loss‚2.6499800256320407 /loss/FBt4/detection_loss‚0.5379543357944869 % _timestamp‚1.7884044116816287e+09 -loss/FRt3/detection_loss‚2.12599079310894 0loss/SRqqt2/detection_loss‚2.476024080406536 1loss/SRqqt3/detection_loss‚1.8003790248185396¢Sa^Í)Ø 0loss/FBt1/assignment_loss‚0.3011743876617402 .loss/FRt4/detection_loss‚1.666017865606894 -loss/FBt3/detection_loss‚3.24802266061306 &loss/total_loss‚42.594033677051044 _runtime‚9036.318391726 _step‚950 0loss/FRt2/assignment_loss‚3.7539447405759026 0loss/FRt3/assignment_loss‚2.3860227251052857 /loss/FRt1/detection_loss‚2.1167276748951465 2loss/SRqqt2/assignment_loss‚1.9749977945378332 .loss/FRt2/detection_loss‚2.873834443442962 1loss/SRqqt4/detection_loss‚0.9306231113698553 .loss/FBt1/detection_loss‚2.492834687232971 epoch‚9 /loss/FRt1/assignment_loss‚3.230871613989485 1loss/SRqqt1/assignment_loss‚1.309479199046659 2loss/SRqqt3/assignment_loss‚1.2801193727646023 2loss/SRqqt4/assignment_loss‚1.0508179530665718 0loss/FBt3/assignment_loss‚0.4787112697315479 1loss/FBt4/assignment_loss‚0.28197732746761334 1loss/SRqqt1/detection_loss‚1.9244119777928952 trainer/global_step‚11549 0loss/FRt4/assignment_loss‚1.4123392167190711 1loss/FBt2/assignment_loss‚0.29077739567040745 /loss/FBt2/detection_loss‚2.6499800256320407 /loss/FBt4/detection_loss‚0.5379543357944869 % _timestamp‚1.7884044116816287e+09 -loss/FRt3/detection_loss‚2.12599079310894 0loss/SRqqt2/detection_loss‚2.476024080406536 1loss/SRqqt3/detection_loss‚1.8003790248185396¶EÏN`Î)‚(:’ ÄÅãÔ¬¸® gpu.0.gpu‚31.0 gpu.0.memory‚1, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚148.016' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.603199999999998 gpu.0.smClock‚1410 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0NÝû•Ï)‚(:Š  ÄÅãÔóŒÄ° disk.drbd2080.in‚0 disk.drbd1716.out‚0" proc.memory.rssMB‚ 2694.4921875 disk.sda3.in‚ 0.06640625& disk.nvme0n1.out‚41264.77978515625 disk.sda.out‚ 8101.48828125 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.drbd2266.in‚0 disk.rbd0.in‚ 0.0546875 disk.dm-1.in‚ 0.06640625# memory_percent‚97.5980281829834* disk./.usagePercent‚15.760488659398728 disk.drbd2266.out‚0 disk.dm-0.in‚ 416.7421875 disk.sda3.out‚ 8101.48828125 disk.rbd1.in‚0! disk.nvme0c0n1.in‚ 416.7421875 disk.sda.in‚ 0.06640625) proc.memory.percent‚16.44587516784668 network.recv‚ 3.208755e+06% disk./.usageGB‚234.84267044067383 disk.rbd1.out‚ 24.7685546875 network.sent‚ 2.8766815e+07 disk.rbd0.out‚ 1.0986328125# disk.dm-0.out‚41264.77978515625 disk.nvme0n1.in‚ 416.7421875 disk.dm-1.out‚ 8101.48828125 cpu‚25.362680746552375 proc.cpu.threads‚17 disk.drbd2265.in‚0 disk.drbd2265.out‚0 disk.sda1.out‚0 disk.sda2.out‚0' proc.memory.availableMB‚ 393.5390625 disk.drbd2080.out‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚41264.77978515625†ºÌÐ)‚(:… ÓÅãÔßÀÑ® gpu.0.gpu‚24.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚108.69' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚21.738 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ø•îü—Ñ)‚(:Œ  ÓÅãÔÝǧ± disk.dm-1.in‚ 0.06640625# disk.dm-0.out‚41266.37255859375 disk.sda3.out‚ 8101.87109375 proc.cpu.threads‚17* disk./.usagePercent‚15.760488659398728 disk.sda1.in‚0 disk.sda3.in‚ 0.06640625 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.rbd0.in‚ 0.0546875 disk.dm-0.in‚ 416.7421875 disk.rbd1.out‚ 24.7685546875! disk.nvme0c0n1.in‚ 416.7421875 disk.nvme0n1.in‚ 416.7421875 disk.drbd2080.out‚0 disk.rbd1.in‚0( disk.nvme0c0n1.out‚41266.37255859375 disk.dm-1.out‚ 8101.87109375 disk.drbd2265.out‚0) proc.memory.percent‚16.44587516784668 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0( proc.memory.availableMB‚ 393.78515625 network.recv‚ 3.215297e+06% disk./.usageGB‚234.84267044067383 disk.sda1.out‚0 disk.sda2.out‚0 disk.drbd2080.in‚0$ memory_percent‚97.59652614593506" proc.memory.rssMB‚ 2694.4921875 network.sent‚ 2.8875695e+07 disk.rbd0.out‚ 1.0986328125& disk.nvme0n1.out‚41266.37255859375 disk.sda.in‚ 0.06640625 disk.sda.out‚ 8101.87109375 cpu‚25.362187459449135woA2Ò)- _wandbruntime‚9071  _runtime‚9071ÖëºÎÉÓ)à trainer/global_step‚11599 % _timestamp‚1.7884044466853085e+09 _runtime‚ 9071.32213674 _step‚951 ' lr-AdamW/pg1‚1.0783235508287205e-06 ' lr-AdamW/pg2‚1.0783235508287205e-06bÔxµÎÔ)È ' lr-AdamW/pg1‚1.0783235508287205e-06 ' lr-AdamW/pg2‚1.0783235508287205e-06 trainer/global_step‚11599 % _timestamp‚1.7884044466853085e+09 _runtime‚ 9071.32213674 _step‚951·Œ®+2Õ)- _wandbruntime‚9071  _runtime‚9071³§J02#Ö)èO A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16361294037504 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚327033854488576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89266906 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 % _timestamp‚1.7884044466859965e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚327414930079826 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65718678 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚346516634115072 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚343394961523200 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 trainer/global_step‚11599 _step‚952 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 387458007 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 238718976 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 152926869 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 387437967 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚343394961523200 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65719203 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚311100972989440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89266904 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16645768957440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚343395148526080 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16361125037568 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 234531136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16313974129746 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 234511098 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚327033836485632 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 234511098 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚311100955950080 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16361125037568 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚343395148526080 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16645554178560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚327415112240590 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 234531136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _runtime‚9071.322979009 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚327033854488576 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16361294037504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 152926871 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚343394961523200 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚65719203 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚311100972989440 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89266904 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16645768957440 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚343395148526080 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16361125037568 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 234531136 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16313974129746 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 234511098 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚327033836485632 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 234511098 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚311100955950080 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16361125037568 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚343395148526080 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16645554178560 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚327415112240590 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚527 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 234531136 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 _runtime‚9071.322979009 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚327033854488576 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16361294037504 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 152926871 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚327033854488576 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89266906 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 % _timestamp‚1.7884044466859965e+09 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚327414930079826 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65718678¸Zgv2Ø)- _wandbruntime‚9071  _runtime‚9071!L\6ø&Ù)òM I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 387470413 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154998629 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16315528415902 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 234550949 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 387490479 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚311126899132416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16362518175744 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387490479 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚327061105850368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚327442262426706 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚363192728586240 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89274281 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16315363294290 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152939528 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7884044473506486e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152939530 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89274283 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 234530885 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16362518175744 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152939530 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65724868 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚327061105850368 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16647181522944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚327442444587678 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 _step‚953 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚343423624026112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚346545523123200 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 trainer/global_step‚11599 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚343423811042304 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚327061123853312 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16362687188992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚9071.987848889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚346545547063296 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16646966757376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 387470413 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚343423811042304 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65724348 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 154999151 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚363192489880576 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152939528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚327061123853312 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 154998629 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16315528415902 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 234550949 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 387490479 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚311126899132416 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16362518175744 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 387490479 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚520 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚327061105850368 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚327442262426706 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚363192728586240 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89274281 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16315363294290 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 152939528 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 % _timestamp‚1.7884044473506486e+09 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 152939530 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89274283 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 234530885 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16362518175744 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 152939530 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65724868 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚327061105850368 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16647181522944 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚327442444587678 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 _step‚953 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚343423624026112 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚346545523123200 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 trainer/global_step‚11599 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚343423811042304 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚327061123853312 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16362687188992 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 _runtime‚9071.987848889 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚346545547063296 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16646966757376 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚522 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 387470413 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚343423811042304 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚65724348 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 154999151 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚363192489880576 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 152939528 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚327061123853312 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 EÞà)‚(:’ ñÅãÔ姮 gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚147.776' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.555200000000003 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0Iw˜á)‚(:  ñÅãÔÒ㎲ disk.sda2.in‚0 network.recv‚ 3.225117e+06 disk.drbd1716.out‚0 disk.sda3.out‚ 8103.04296875 disk.rbd1.out‚ 24.95703125( proc.memory.availableMB‚ 482.26171875 network.sent‚ 2.8989531e+07 disk.drbd2266.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚ 1.0986328125 disk.dm-0.in‚ 416.76953125$ memory_percent‚97.05650806427002 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.rbd0.in‚ 0.0546875 disk.sda3.in‚ 0.06640625" disk.nvme0c0n1.in‚ 416.76953125 disk.sda2.out‚0" proc.memory.rssMB‚ 2604.2734375 disk.dm-1.in‚ 0.06640625# disk.dm-0.out‚41364.13427734375 disk.sda.out‚ 8103.04296875 disk.sda1.out‚0 disk.rbd1.in‚0 disk.sda.in‚ 0.06640625 proc.cpu.threads‚17* disk./.usagePercent‚15.894968137097482 disk.drbd2265.out‚0 disk.sda1.in‚0( disk.nvme0c0n1.out‚41364.13427734375 disk.nvme0n1.in‚ 416.76953125* proc.memory.percent‚15.895223617553711 cpu‚25.361127913979722$ disk./.usageGB‚236.8465118408203 disk.dm-1.out‚ 8103.04296875& disk.nvme0n1.out‚41364.13427734375àr`±â)‚(:’ €ÆãÔƒéú­ gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚146.131' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.226200000000002 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0vsˆŒã)‚(:…  €ÆãÔ‘€¿±$ memory_percent‚97.05512523651123# proc.memory.rssMB‚ 2604.27734375 proc.cpu.threads‚17 disk.nvme0n1.in‚ 419.8828125 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.rbd0.in‚ 0.0546875 disk.rbd0.out‚ 1.0986328125$ disk./.usageGB‚236.1688690185547& disk.nvme0n1.out‚42626.21240234375 disk.dm-1.in‚ 0.06640625 disk.drbd2266.in‚0 disk.drbd2265.in‚0( proc.memory.availableMB‚ 482.48828125* proc.memory.percent‚15.895247459411621* disk./.usagePercent‚15.849490958715037 disk.sda.in‚ 0.06640625 disk.drbd1716.out‚0 disk.dm-0.in‚ 419.8828125 disk.sda1.in‚0 disk.sda2.in‚0 disk.rbd1.out‚ 25.01953125 network.recv‚ 3.22919e+06! disk.nvme0c0n1.in‚ 419.8828125 disk.dm-1.out‚ 8103.7109375# disk.dm-0.out‚42626.21240234375 disk.sda3.in‚ 0.06640625 network.sent‚ 2.8995522e+07 disk.sda.out‚ 8103.7109375 disk.sda2.out‚0 disk.drbd2080.out‚0 disk.sda1.out‚0 disk.sda3.out‚ 8103.7109375 disk.rbd1.in‚0 cpu‚25.3606203998665( disk.nvme0c0n1.out‚42626.21240234375 disk.drbd2266.out‚0 disk.drbd2080.in‚0ÑÅΨ2ä)- _wandbruntime‚9107  _runtime‚9107ÄåÁQÈå) & lr-AdamW/pg1‚8.701696195168576e-07 & lr-AdamW/pg2‚8.701696195168576e-07 trainer/global_step‚11649 % _timestamp‚1.7884044823716063e+09 _runtime‚9107.008445956 _step‚955ˆØÍæ)Ç & lr-AdamW/pg2‚8.701696195168576e-07 trainer/global_step‚11649 % _timestamp‚1.7884044823716063e+09 _runtime‚9107.008445956 _step‚955 & lr-AdamW/pg1‚8.701696195168576e-07»¥²Wâ2ç)- _wandbruntime‚9107  _runtime‚9107Œpnì'è)æO WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16383445948154 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚344828288085504 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚344828101082624 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 235503800 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚328781743175798 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚328397304722432 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 235523838 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153559821 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153559821 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 _step‚956 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚328397304722432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚328397322725376 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16716638663168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚344828288085504 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16430796360192 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 235523838 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155624718 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65988964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚347962257586176 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16383611069558 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 389083659 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 389063619 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153559819 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153559819 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 155625262 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16430965360128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89635756 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚364678685664768 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚344828101082624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚9107.009345824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89635754 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16716428078592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚328781561015034 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16430796360192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544jv`ñ'é)ëO UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16430796360192 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚544 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16383445948154 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚344828288085504 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚344828101082624 JDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 235503800 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 234524672 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚328781743175798 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚328397304722432 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 235523838 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 153559821 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 153559821 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 _step‚956 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚328397304722432 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚328397322725376 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16716638663168 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚344828288085504 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16430796360192 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 235523838 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 155624718 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚65988964 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚347962257586176 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16383611069558 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 389083659 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 389063619 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 153559819 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 153559819 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 155625262 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16430965360128 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚89635756 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚364678685664768 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚344828101082624 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 _runtime‚9107.009345824 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚89635754 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16716428078592 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚328781561015034 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 210584576 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199¼'Fœ2ê)- _wandbruntime‚9107  _runtime‚9107™”Gë)ñM WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚344856950601728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚364709254777856 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153572478 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚344856763585536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89643131 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16718060303360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚328424574087168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65994807 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16432189498368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244997120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚328808893361914 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 % _timestamp‚1.7884044830457168e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚328424592090112 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚328424574087168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 235523587 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚328424592090112 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PÞbal_step‚11649 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 153572480 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16384835112698 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚524 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16717839246336 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 389096065 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 235543651 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89643133 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16385000234310 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 389116131 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚347991170534400 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16432358511616 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155637416 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚364709009780736 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 235543651 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚957 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16432358511616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _runtime‚ 9107.68313449 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389116131 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224Ò¡›ü&ì)öM VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚65994807 B6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16432189498368 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 244997120 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚328808893361914 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 % _timestamp‚1.7884044830457168e+09 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚328424592090112 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚328424574087168 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 235523587 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚328424592090112 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 PDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 389096065 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚522 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 235543651 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚89643133 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16385000234310 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 389116131 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚347991170534400 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16432358511616 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 155637416 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚364709009780736 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 EDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 235543651 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 _step‚957 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16432358511616 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 _runtime‚ 9107.68313449 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 389116131 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚344856950601728 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚364709254777856 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 153572478 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚344856763585536 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚89643131 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16718060303360 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚328424574087168 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248½܇q†2í)- _wandbruntime‚9107  _runtime‚9107§¦¹Tî)Î /loss/FBt4/detection_loss‚0.8343206647938738 _step‚958 /loss/FRt3/assignment_loss‚2.727780572573344 -loss/FRt1/detection_loss‚2.28821685851789 /loss/FRt2/assignment_loss‚4.020707875490189 2loss/SRqqt2/assignment_loss‚1.9155343785558059 1loss/SRqqt3/assignment_loss‚1.280497518615151 .loss/FRt3/detection_loss‚2.355915457010269 /loss/FRt4/detection_loss‚1.8415094131293397 .loss/FBt3/detection_loss‚3.321360372006893 trainer/global_step‚11649 _runtime‚9107.684098779 /loss/FRt1/assignment_loss‚2.967380601985782 /loss/FRt4/assignment_loss‚1.522360975543658 -loss/FRt2/detection_loss‚3.06761477461883 1loss/FBt1/assignment_loss‚0.19889867367657224 0loss/SRqqt2/detection_loss‚2.765650741432024 -loss/FBt1/detection_loss‚2.14804194523738 &loss/total_loss‚43.833813069083035 1loss/FBt3/assignment_loss‚0.23643881462861316 2loss/SRqqt1/assignment_loss‚1.5383859698884879 2loss/SRqqt4/assignment_loss‚0.7829125594634276 1loss/FBt2/assignment_loss‚0.13954358315095305 0loss/SRqqt4/detection_loss‚1.227017259483154 /loss/FBt2/detection_loss‚3.0528344937733243 epoch‚9 $ _timestamp‚1.788404483047382e+09 1loss/FBt4/assignment_loss‚0.24198421020915994 0loss/SRqqt1/detection_loss‚1.581033986199785 1loss/SRqqt3/detection_loss‚1.7778713690991303[Wß•Yï)Ó /loss/FBt2/detection_loss‚3.0528344937733243 epoch‚9 $ _timestamp‚1.788404483047382e+09 1loss/FBt4/assignment_loss‚0.24198421020915994 0loss/SRqqt1/detection_loss‚1.581033986199785 1loss/SRqqt3/detection_loss‚1.7778713690991303 /loss/FBt4/detection_loss‚0.8343206647938738 _step‚958 /loss/FRt3/assignment_loss‚2.727780572573344 -loss/FRt1/detection_loss‚2.28821685851789 /loss/FRt2/assignment_loss‚4.020707875490189 2loss/SRqqt2/assignment_loss‚1.9155343785558059 1loss/SRqqt3/assignment_loss‚1.280497518615151 .loss/FRt3/detection_loss‚2.355915457010269 /loss/FRt4/detection_loss‚1.8415094131293397 .loss/FBt3/detection_loss‚3.321360372006893 trainer/global_step‚11649 _runtime‚9107.684098779 /loss/FRt1/assignment_loss‚2.967380601985782 /loss/FRt4/assignment_loss‚1.522360975543658 -loss/FRt2/detection_loss‚3.06761477461883 1loss/FBt1/assignment_loss‚0.19889867367657224 0loss/SRqqt2/detection_loss‚2.765650741432024 -loss/FBt1/detection_loss‚2.14804194523738 &loss/total_loss‚43.833813069083035 1loss/FBt3/assignment_loss‚0.23643881462861316 2loss/SRqqt1/assignment_loss‚1.5383859698884879 2loss/SRqqt4/assignment_loss‚0.7829125594634276 1loss/FBt2/assignment_loss‚0.13954358315095305 0loss/SRqqt4/detection_loss‚1.227017259483154¾aÛ;ð)‚(:… ÆãÔ×Å¢® gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚140.51' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.102 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0‘æXF—ñ)‚(:Œ  ÆãÔ³‹ü± disk.rbd0.in‚ 0.0546875 disk.dm-0.in‚ 420.91015625 cpu‚25.360108122013425 disk.sda.out‚ 8104.13671875 disk.drbd1716.out‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.rbd0.out‚ 1.0986328125( disk.nvme0c0n1.out‚42632.73779296875# proc.memory.rssMB‚ 2604.27734375 proc.cpu.threads‚17 disk.drbd2265.out‚0# disk.dm-0.out‚42632.73779296875 disk.sda1.out‚0 disk.sda3.out‚ 8104.13671875$ memory_percent‚97.05536365509033* proc.memory.percent‚15.895247459411621 disk.rbd1.out‚ 25.01953125( proc.memory.availableMB‚ 482.44921875 disk.sda1.in‚0 network.recv‚ 3.2337e+06 disk.dm-1.in‚ 0.06640625 disk.sda3.in‚ 0.06640625 network.sent‚ 2.9001803e+07 disk.nvme0n1.in‚ 420.91015625 disk.sda.in‚ 0.06640625 disk.drbd2266.out‚0" disk.nvme0c0n1.in‚ 420.91015625$ disk./.usageGB‚236.1719970703125& disk.nvme0n1.out‚42632.73779296875 disk.sda2.in‚0 disk.sda2.out‚0 disk.dm-1.out‚ 8104.13671875 disk.drbd2266.in‚0 disk.drbd2080.in‚0 disk.rbd1.in‚0* disk./.usagePercent‚15.849700884893114 disk.drbd1716.in‚0mÉ´ò)‚(:’ žÆãԃ烮 gpu.0.gpu‚24.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚132.419' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚26.483800000000002 gpu.0.smClock‚1335 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0P°Lšó)‚(:  žÆãÔèúÊ° disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2265.out‚0 disk.sda1.in‚0# proc.memory.rssMB‚ 2604.28515625 proc.cpu.threads‚17 network.recv‚ 3.239162e+06% disk./.usageGB‚236.17140579223633" disk.nvme0c0n1.in‚ 420.91015625* proc.memory.percent‚15.895295143127441 disk.dm-0.in‚ 420.91015625& disk.nvme0n1.out‚42633.20654296875 network.sent‚ 2.9109885e+07 disk.sda1.out‚0 disk.rbd1.out‚ 25.0517578125 disk.nvme0n1.in‚ 420.91015625 disk.sda2.in‚0 disk.drbd2080.in‚0 disk.sda3.in‚ 0.06640625* disk./.usagePercent‚15.849661203725308 disk.drbd1716.in‚0 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.rbd0.in‚ 0.0546875 disk.rbd0.out‚ 1.0986328125# disk.dm-0.out‚42633.20654296875( disk.nvme0c0n1.out‚42633.20654296875 disk.sda.in‚ 0.06640625 disk.dm-1.out‚ 8104.69921875 disk.drbd2080.out‚0 disk.sda3.out‚ 8104.69921875 disk.rbd1.in‚0$ memory_percent‚97.05691337585449' proc.memory.availableMB‚ 482.1953125 cpu‚25.35958093534148 disk.sda.out‚ 8104.69921875 disk.sda2.out‚0ƒü2ô)- _wandbruntime‚9142  _runtime‚9142â­ èÈõ) & lr-AdamW/pg1‚6.842357901870163e-07 & lr-AdamW/pg2‚6.842357901870163e-07 trainer/global_step‚11699 % _timestamp‚1.7884045181541543e+09 _runtime‚9142.790994591 _step‚959Ø~Š7Íö)Ç & lr-AdamW/pg2‚6.842357901870163e-07 trainer/global_step‚11699 % _timestamp‚1.7884045181541543e+09 _runtime‚9142.790994591 _step‚959 & lr-AdamW/pg1‚6.842357901870163e-07¿÷‹W2÷)- _wandbruntime‚9142  _runtime‚9142“?í'ø)çO _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16453082887966 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 390689271 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚346261240642048 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16500636682752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16500636682752 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16452917766562 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154192769 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 156265613 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚366195446116352 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 156266175 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚329760790962176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66261569 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚330148191950242 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚329760772959232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90004604 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 236516540 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16500467682816 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚329760772959232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚349407904997376 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 390709311 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚11699 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154192771 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 236496502 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66261009 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚346261427644928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚313695274183680 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 236516540 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚562 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚560 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚346261240642048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90004606 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚313695291223040 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154192771 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚330148374111006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16787517178880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚9142.791962497 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 236496502 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 $ _timestamp‚1.788404518154903e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚329760790962176 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16500467682816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \HDeviceStatsMonitor.on_train_batch_star÷Á?ÔX"t/allocated_bytes.large_pool.freed‚329760772959232 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚349407904997376 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 390709311 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 trainer/global_step‚11699 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 154192771 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 236496502 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66261009 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚346261427644928 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚313695274183680 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 236516540 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚562 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚560 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚346261240642048 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90004606 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚313695291223040 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 154192771 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚330148374111006 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16787517178880 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 _runtime‚9142.791962497 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 236496502 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 $ _timestamp‚1.788404518154903e+09 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚329760790962176 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16500467682816 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 BDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚346261240642048 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16500636682752 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16500636682752 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 208487424 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16452917766562 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154192769 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 156265613 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚366195446116352 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 156266175 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚329760790962176 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66261569 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚330148191950242 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚329760772959232 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90004604À¥t±;2ú)- _wandbruntime‚9143  _runtime‚9143€Æ]åø&û)òM T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚346290090161152 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16501860820992 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚366225780297216 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16788938411520 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16788725743104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚329788060326912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90011983 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66266868 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16454306931106 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚346289903144960 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 % _timestamp‚1.7884045188299775e+09 _runtime‚9143.467400551 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16502029834240 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154205430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154205428 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154205428 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚329788060326912 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 trainer/global_step‚11699 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 _step‚961 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 236536353 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 236536353 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 236516289 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156278320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚346290090161152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 390741783 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66266339 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 390721717 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚330175706458094 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 236516289 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 156278851 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90011981 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416¦Ãç³ý&ü)÷M M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 236516289 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 156278851 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90011981 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚531 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚346290090161152 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16501860820992 FDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚366225780297216 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16788938411520 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16788725743104 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚329788060326912 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90011983 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66266868 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16454306931106 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚346289903144960 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 % _timestamp‚1.7884045188299775e+09 _runtime‚9143.467400551 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16502029834240 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154205430 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 154205428 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154205428 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚329788060326912 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 trainer/global_step‚11699 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 _step‚961 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 236536353 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 236536353 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 236516289 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156278320 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚346290090161152 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 390741783 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66266339 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 390721717 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚330175706458094Á^µÛ!2ý)- _wandbruntime‚9143  _runtime‚9143âéÁ\þ)Ö _step‚962 0loss/FRt1/assignment_loss‚3.1404892864980196 1loss/SRqqt3/detection_loss‚1.6938210877321533 /loss/FBt1/detection_loss‚3.0857041120529174 1loss/SRqqt3/assignment_loss‚1.470541831839956 1loss/SRqqt2/assignment_loss‚2.404337696073686 /loss/FRt2/detection_loss‚2.6551271066433046 /loss/FRt4/detection_loss‚1.7561841492765937 /loss/FBt2/detection_loss‚2.7954823757920946 0loss/SRqqt4/assignment_loss‚1.01366397684467 1loss/FBt1/assignment_loss‚0.31039076838640084 1loss/FBt3/assignment_loss‚0.24346466511486256 -loss/FRt3/detection_loss‚2.35256189070642 1loss/SRqqt1/detection_loss‚1.8010722013576976 1loss/SRqqt2/detection_loss‚2.4487572294824265 /loss/FRt2/assignment_loss‚3.780797103556191 0loss/FRt4/assignment_loss‚1.4849274960057488 1loss/SRqqt4/detection_loss‚1.0355445850586547 %loss/total_loss‚43.41793262763085 0loss/FRt3/assignment_loss‚2.3930707082152365 0loss/FBt2/assignment_loss‚0.3593392747263075 /loss/FBt4/detection_loss‚0.9646266498990977 2loss/SRqqt1/assignment_loss‚1.5260314358538896 /loss/FBt3/detection_loss‚2.3918701350688933 epoch‚9 trainer/global_step‚11699 _runtime‚9143.468455682 1loss/FBt4/assignment_loss‚0.19535499809801612 /loss/FRt1/detection_loss‚2.1147718633476056 % _timestamp‚1.7884045188316743e+09n[šÑaÿ)Û 1loss/SRqqt2/assignment_loss‚2.404337696073686 /loss/FRt2/detection_loss‚2.6551271066433046 /loss/FRt4/detection_loss‚1.7561841492765937 /loss/FBt2/detection_loss‚2.7954823757920946 0loss/SRqqt4/assignment_loss‚1.01366397684467 1loss/FBt1/assignment_loss‚0.31039076838640084 1loss/FBt3/assignment_loss‚0.24346466511486256 -loss/FRt3/detection_loss‚2.35256189070642 1loss/SRqqt1/detection_loss‚1.8010722013576976 1loss/SRqqt2/detection_loss‚2.4487572294824265 /loss/FRt2/assignment_loss‚3.780797103556191 0loss/FRt4/assignment_loss‚1.4849274960057488 1loss/SRqqt4/detection_loss‚1.0355445850586547 %loss/total_loss‚43.41793262763085 0loss/FRt3/assignment_loss‚2.3930707082152365 0loss/FBt2/assignment_loss‚0.3593392747263075 /loss/FBt4/detection_loss‚0.9646266498990977 2loss/SRqqt1/assignment_loss‚1.5260314358538896 /loss/FBt3/detection_loss‚2.3918701350688933 epoch‚9 trainer/global_step‚11699 _runtime‚9143.468455682 1loss/FBt4/assignment_loss‚0.19535499809801612 /loss/FRt1/detection_loss‚2.1147718633476056 % _timestamp‚1.7884045188316743e+09 _step‚962 0loss/FRt1/assignment_loss‚3.1404892864980196 1loss/SRqqt3/detection_loss‚1.6938210877321533 /loss/FBt1/detection_loss‚3.0857041120529174 1loss/SRqqt3/assignment_loss‚1.470541831839956Âùú<€*‚(:’ ­ÆãÔ彺® gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚138.866' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.773200000000003 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0œ4©¶*‚(:  ­ÆãԈ…± disk.rbd1.out‚ 25.1455078125 disk.sda2.in‚0 disk.dm-1.out‚ 8105.23828125 disk.drbd2266.in‚0 disk.drbd2080.in‚0" disk.nvme0c0n1.in‚ 420.91015625 disk.sda.out‚ 8105.23828125" proc.memory.rssMB‚ 2604.2890625 network.sent‚ 2.9116038e+07* disk./.usagePercent‚15.849661203725308 disk.rbd0.out‚ 1.0986328125 disk.dm-0.in‚ 420.91015625 disk.sda3.in‚ 0.06640625 disk.nvme0n1.in‚ 420.91015625& disk.nvme0n1.out‚42633.67919921875 disk.dm-1.in‚ 0.06640625 disk.drbd2266.out‚0# disk.dm-0.out‚42633.67919921875 disk.sda3.out‚ 8105.23828125 disk.sda.in‚ 0.06640625* proc.memory.percent‚15.89531898 .oÇâ4985352 network.recv‚ 3.243602e+06% disk./.usageGB‚236.17140579223633 disk.rbd1.in‚0 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚ 0.0546875 disk.sda2.out‚0# memory_percent‚97.0555305480957( disk.nvme0c0n1.out‚42633.67919921875 cpu‚25.359044910266096 proc.cpu.threads‚17 disk.sda1.in‚0& proc.memory.availableMB‚ 482.421875 disk.drbd1716.in‚0 disk.drbd2265.in‚0 disk.sda1.out‚0 ™‚*‚(:’ ¼ÆãÔÍÜ’® gpu.0.gpu‚38.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚145.065' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.012999999999998 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÚPòa’ƒ*‚(:‡  ¼ÆãÔ€ü°" proc.memory.rssMB‚ 2604.2890625 disk.rbd0.out‚ 1.0986328125" disk.dm-0.out‚42634.3759765625 disk.sda3.in‚ 0.06640625* disk./.usagePercent‚15.849662995778047$ disk./.usageGB‚236.1714324951172 disk.sda.in‚ 0.06640625 disk.drbd1716.in‚0 network.sent‚ 2.9224414e+07 disk.sda2.out‚0 disk.drbd1716.out‚0 disk.drbd2266.in‚0 disk.rbd0.in‚ 0.0546875' disk.nvme0c0n1.out‚42634.3759765625 disk.nvme0n1.in‚ 420.91015625( proc.memory.availableMB‚ 482.42578125 disk.drbd2266.out‚0 disk.sda3.out‚ 8105.7734375 network.recv‚ 3.24952e+06 disk.dm-0.in‚ 420.91015625 disk.sda1.out‚0% disk.nvme0n1.out‚42634.3759765625 disk.sda.out‚ 8105.7734375 disk.drbd2265.in‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0* proc.memory.percent‚15.895318984985352 disk.sda2.in‚0 disk.dm-1.out‚ 8105.7734375 disk.drbd2265.out‚0 cpu‚25.358568114548888 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 420.91015625 proc.cpu.threads‚17 disk.dm-1.in‚ 0.06640625 disk.rbd1.in‚0 disk.rbd1.out‚ 25.2080078125# memory_percent‚97.0555067062378HøÁ2„*- _wandbruntime‚9178  _runtime‚9178$mÄÆ…*À _runtime‚9178.561347878 _step‚963 % lr-AdamW/pg1‚5.20560811170553e-07 % lr-AdamW/pg2‚5.20560811170553e-07 trainer/global_step‚11749 % _timestamp‚1.7884045539244158e+09’ù5”ˆ*Å % lr-AdamW/pg2‚5.20560811170553e-07 trainer/global_step‚11749 % _timestamp‚1.7884045539244158e+09 _runtime‚9178.561347878 _step‚963 % lr-AdamW/pg1‚5.20560811170553e-07Ãb?a‹2‡*- _wandbruntime‚9178  _runtime‚9178çÔjî'ˆ*èO LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 156903415 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16570308005376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16522554706374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚331124241196032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66529959 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 % _timestamp‚1.7884045539251356e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16858384271872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 392334963 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚347694567204352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚314992433300480 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16522389584970 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 237489204 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16570308005376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90373454 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚367711960620544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚331514822885450 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚314992450339840 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16570139005440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154825719 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚331124259198976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66529440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16858179978752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚331515005046214 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 trainer/global_step‚11749 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚350853552408576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204293120 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚347694567204352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 237509242 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚331124259198976 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚350853576348672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚347694380201472 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _runtime‚9178.562189295 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 392314923 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 237489204 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248“.bçó'‰*íO VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90373456 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 JDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚331514822885450 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚314992450339840 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16570139005440 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 154825719 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚331124259198976 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚66529440 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚16858179978752 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚331515005046214 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 trainer/global_step‚11749 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚350853552408576 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 204293120 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚347694567204352 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 237509242 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚331124259198976 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚350853576348672 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚347694380201472 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 _runtime‚9178.562189295 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 392314923 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 237489204 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 156903415 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16570308005376 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16522554706374 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚331124241196032 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚66529959 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 % _timestamp‚1.7884045539251356e+09 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16858384271872 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 392334963 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚347694567204352 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 228233216 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚314992433300480 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16522389584970 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 237489204 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16570308005376 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90373454 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚367711960620544 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156915586 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16859597466624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16571532143616 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 392347369 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154838380 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16571532143616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154838380 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16571701156864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 392347369 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 156916088 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16523778749514 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16571701156864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚331542155232330 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚502 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚347723229720576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚367742062823424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚331151510560768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66534755 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16859805940736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚350882489296896 trainer/global_step‚11749 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 237529055 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 392367435 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚315018376482816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16523943871126 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 392367435 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66535255 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚331151528563712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚347723229720576 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚347723042704384 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚331151510560768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚331542337393302 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚9179.230564488 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 237508991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚350882465356800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚965 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90380831 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90380833 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚331151528563712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚367742295237632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_en3¸¦^d/segment.small_pool.current‚199 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0.ÍîŠý&Œ*÷M ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 % _timestamp‚1.7884045545931315e+09 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 154838378 EDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 156915586 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16859597466624 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16571532143616 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 392347369 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 154838380 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16571532143616 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 154838380 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16571701156864 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 392347369 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 156916088 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16523778749514 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16571701156864 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚331542155232330 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚502 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚347723229720576 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚367742062823424 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚331151510560768 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66534755 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16859805940736 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚350882489296896 trainer/global_step‚11749 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 237529055 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 392367435 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚315018376482816 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16523943871126 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 392367435 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚66535255 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚331151528563712 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚347723229720576 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚347723042704384 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚331151510560768 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚331542337393302 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 _runtime‚9179.230564488 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 237508991 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚350882465356800 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 _step‚965 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90380831 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90380833 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚331151528563712 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚367742295237632 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 237508991 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 PDeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90742306 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚349127519760896 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 238481906 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 157543828 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16639810328064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚11799 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚332487727435776 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16929308963840 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚349127706763776 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 _step‚968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 393940575 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 238501944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚521 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90742304 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16639810328064 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349127519760896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155458669 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚316289609456640 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚369228294004736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚352299223759872 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚332881635981422 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚332487709432832 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401x¼òÛä›*íO \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚316289592417280 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚90742306 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚349127519760896 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 238481906 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 157543828 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16639810328064 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 trainer/global_step‚11799 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚332487727435776 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚16929308963840 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚349127706763776 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 _step‚968 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 393940575 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 238501944 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.curgžrent‚521 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚90742304 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16639810328064 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚349127519760896 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 155458669 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚316289609456640 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚369228294004736 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚352299223759872 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚332881635981422 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚332487709432832 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16639979328000 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 155458669 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 393973021 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 238521757 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚352328136708096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚332908786167538 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 238501693 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 238501693 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 393973021 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66806358 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 % _timestamp‚1.7884045897684991e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 393993087 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 393993087 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16641372479488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90749683 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚316315535599616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚332514978797568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90749681 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16930718261760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16641372479488 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155471328 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155471330 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚332908968328510 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚349156369280000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 _step‚969 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157556039 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221928448 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16593415689534 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚332514978797568 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚502 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚352328112768000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349156369280000 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16641203466240 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 trainer/global_step‚11799 bSÈý&ž*÷M L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 157556039 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 221928448 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16593415689534 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚332514978797568 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚502 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚352328112768000 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚349156369280000 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16641203466240 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 trainer/global_step‚11799 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 238521757 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚504 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚16930520273408 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚316315552638976 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚332514996800512 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 F6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 393973021 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 238521757 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚352328136708096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚332908786167538 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 238501693 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 238501693 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 393973021 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚66806358 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 % _timestamp‚1.7884045897684991e+09 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 393993087 B6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 393993087 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16641372479488 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚90749683 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚316315535599616 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚332514978797568 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚90749681 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚16930718261760 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16641372479488 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 155471328 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 155471330 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚332908968328510 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚349156369280000 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 _step‚969 PçÞʧ*Ä ' lr-AdamW/pg1‚2.6011919673956037e-07 ' lr-AdamW/pg2‚2.6011919673956037e-07 trainer/global_step‚11849 % _timestamp‚1.7884046246262171e+09 _runtime‚9249.263069747 _step‚971‹üºµÏ¨*É trainer/global_step‚11849 % _timestamp‚1.7884046246262171e+09 _runtime‚9249.263069747 _step‚971 ' lr-AdamW/pg1‚2.6011919673956037e-07 ' lr-AdamW/pg2‚2.6011919673956037e-07ËÆË×,2©*- _wandbruntime‚9249  _runtime‚9249~ý+!ª*çO @8DeviceStatsMonitor.on_tÒdŸÌ'rain_batch_start/segment.all.peak‚416 _runtime‚9249.263941782 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67070480 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚333851195672576 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚317586768573440 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156091619 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 _step‚972 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16661498343190 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚499 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚334248266916630 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 395586267 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156091619 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16709481650688 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 239474608 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67069983 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156091621 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚350560846323200 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚333851195672576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚334248084755866 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚353744847230976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚370745074824192 trainer/global_step‚11849 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚350560846323200 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16709650650624 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 239474608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91111154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚497 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚350560659320320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚333851177669632 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 239494646 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚350560846323200 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚333851195672576 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚334248084755866 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚353744847230976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚370745074824192 trainer/global_step‚11849 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚350560846323200 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 212681728 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16709650650624 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 239474608 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91111154 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚497 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚350560659320320 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚333851177669632 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 236621824 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 239494646 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 395618739 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚350589508839424 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 239494395 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 _step‚973 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚333878465037312 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚474 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚370775169093120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚353773760179200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156104280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚317612694716416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚334275417102746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158193711 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17001408913920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚334275599263718 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 395618739 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156104278 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67075180 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333878465037312 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16710874788864 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚350589321823232 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 158194187 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156104280 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 239514459 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91118533 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17001617388032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 239494395 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚370775401507328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91118531 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16662722386330 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 395598673 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 BDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 208474112 BDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 395618739 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚350589508839424 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 239494395 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 _step‚973 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚333878465037312 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚474 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚370775169093120 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚353773760179200 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156104280 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚317612694716416 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚334275417102746 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 PDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158193711 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17001408913920 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚334275599263718 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 395618739 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 156104278 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67075180 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 RDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.peak‚ 417333248 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚333878465037312 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16710874788864 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚350589321823232 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 158194187 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 156104280 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 239514459 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91118533 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17001617388032 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 239494395 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚370775401507328 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91118531 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16662722386330 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 395598673 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 FDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚351993798879744 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚355190518582272 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67338561 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚351993985882624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚318883910650880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚335214663909376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚335214645906432 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156724569 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16779152973312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67339072 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156724571 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 240487348 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16779152973312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16730805040194 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156724569 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚351993985882624 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 240467310 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 # _timestamp‚1.78840465999282e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 397191879 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156724571 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚335214645906432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚513 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 240467310 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚372261418695680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚355190494642176 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚335614897851838 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚351993798879744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91480006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚351993798879744 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚355190518582272 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67338561 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚351993985882624 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚318883910650880 `LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.allocated‚335214663909376 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚335214645906432 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 156724569 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16779152973312 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67339072 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 156724571 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 240487348 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16779152973312 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16730805040194 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 156724569 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚351993985882624 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 240467310 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 G?DeviceStatgn¨—†sMonitor.on_train_batch_start/segment.small_pool.peak‚199 # _timestamp‚1.78840465999282e+09 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 397191879 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 156724571 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚335214645906432 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚513 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 240467310 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚372261418695680 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚355190494642176 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚335614897851838 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚351993798879744 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91480006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416Ð… ¼*‚(:’ ´ÇãÔŸö§® gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚142.402' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚28.480399999999996 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0½Áዽ*‚(:€  ´ÇãÔ à± disk.drbd2266.in‚0 cpu‚25.354429468756432 network.recv‚ 3.288784e+06$ disk./.usageGB‚236.1728858947754' disk.nvme0c0n1.out‚42638.9287109375% disk.nvme0n1.out‚42638.9287109375 disk.sda2.in‚0 disk.drbd1716.in‚0" disk.dm-0.out‚42638.9287109375 disk.sda1.in‚0 disk.rbd0.out‚ 1.0986328125 disk.sda1.out‚0" disk.nvme0c0n1.in‚ 420.91015625 disk.nvme0n1.in‚ 420.91015625 disk.sda.in‚ 0.06640625 disk.drbd2266.out‚0 disk.drbd2265.in‚0( proc.memory.availableMB‚ 481.88671875 disk.sda2.out‚0 disk.dm-1.in‚ 0.06640625# memory_percent‚97.0587968826294 disk.rbd1.in‚0 disk.sda.out‚ 8110.28125 disk.drbd2080.out‚0 disk.dm-0.in‚ 420.91015625 disk.sda3.in‚ 0.06640625* proc.memory.percent‚15.895318984985352 network.sent‚ 2.9578929e+07) disk./.usagePercent‚15.84976053464859 disk.drbd1716.out‚0 disk.drbd2080.in‚0" proc.memory.rssMB‚ 2604.2890625 disk.drbd2265.out‚0 disk.rbd0.in‚ 0.0546875 disk.sda3.out‚ 8110.28125 proc.cpu.threads‚17 disk.rbd1.out‚ 25.701171875 disk.dm-1.out‚ 8110.28125Aå<ï2¾*- _wandbruntime‚9285  _runtime‚9285leï–ø&¿*òM M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 397224325 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16780546111488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚355219407590400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚352022648398848 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67343951 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚372291984812544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚352022648398848 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚335642230198926 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _runtime‚9285.304675547 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16780715124736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 397224325 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚355219431530496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11899 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16780715124736 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚335241933274112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16732194204738 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 158831842 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚372291744009728 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17072553282048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚318909870872576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 397244391 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚335642048037954 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 240487097 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚352022461382656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚335241915271168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91487383 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚508 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚335241933274112 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156737230 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17072336419328 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 % _timestamp‚1.7884046606671913e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156737228 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 240487097 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158831332 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91487381 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 240507161 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚335241915271168 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16780546111488 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ÷„Õý&À*÷M UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 PDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 397244391 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚335642048037954 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 240487097 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚352022461382656 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚335241915271168 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91487383 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚508 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚335241933274112 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 156737230 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17072336419328 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 % _timestamp‚1.7884046606671913e+09 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 156737228 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 240487097 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 158831332 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91487381 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 240507161 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚335241915271168 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16780546111488 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 397224325 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16780546111488 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚355219407590400 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚352022648398848 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67343951 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚372291984812544 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚352022648398848 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚335642230198926 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 _runtime‚9285.304675547 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16780715124736 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 397224325 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚355219431530496 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 EDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 trainer/global_step‚11899 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16780715124736 FDeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚335241933274112 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16732194204738 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 MCDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.current‚20064 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 158831842 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 XDDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.freed‚372291744009728 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17072553282048 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚318909870872576 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 240802816ÑUPl)2Á*- _wandbruntime‚9285  _runtime‚9285p´¥°YÂ*Ó 0loss/FRt4/assignment_loss‚1.4581385056177776 .loss/FRt2/detection_loss‚3.066178798675537 0loss/SRqqt2/detection_loss‚2.822802874057189 /loss/FBt3/detection_loss‚2.9669464145387923 trainer/global_step‚11899 0loss/FRt2/assignment_loss‚3.9277717344688647 2loss/SRqqt1/assignment_loss‚1.6704819390165901 /loss/FRt1/detection_loss‚2.3925555694934935 0loss/SRqqt1/detection_loss‚1.377739746892263 /loss/FBt2/detection_loss‚2.1687118162711463 2loss/SRqqt3/assignment_loss‚1.4438469956882976 0loss/FBt4/assignment_loss‚0.2321897712768184 /loss/FRt3/detection_loss‚2.6101779974997044 0loss/SRqqt3/detection_loss‚1.727225892415101 /loss/FRt3/assignment_loss‚2.182500660419464 /loss/FBt1/assignment_loss‚0.510374205366033 0loss/FBt2/assignment_loss‚0.5115130462993681 1loss/SRqqt4/detection_loss‚1.0786961610226526 epoch‚9 2loss/SRqqt4/assignment_loss‚0.7783085169824029 _runtime‚9285.305835696 1loss/FBt3/assignment_loss‚0.31010619391287975 /loss/FRt4/detection_loss‚1.7901477644256516 -loss/FBt1/detection_loss‚2.19178673854241 /loss/FBt4/detection_loss‚0.6757474708526084 _step‚978 0loss/FRt1/assignment_loss‚2.9235467079073882 1loss/SRqqt2/assignment_loss‚2.140413442383642 &loss/total_loss‚42.957908964026075 % _timestamp‚1.7884046606690958e+09Ø°^Ã*Ø 1loss/FBt3/assignment_loss‚0.31010619391287975 /loss/FRt4/detection_loss‚1.7901477644256516 -loss/FBt1/detection_loss‚2.19178673854241 /loss/FBt4/detection_loss‚0.6757474708526084 _step‚978 0loss/FRt1/assignment_loss‚2.9235467079073882 1loss/SRqqt2/assignment_loss‚2.140413442383642 &loss/total_loss‚42.957908964026075 % _timestamp‚1.7884046606690958e+09 0loss/FRt4/assignment_loss‚1.4581385056177776 .loss/FRt2/detection_loss‚3.066178798675537 0loss/SRqqt2/detection_loss‚2.822802874057189 /loss/FBt3/detection_loss‚2.9669464145387923 trainer/global_step‚11899 0loss/FRt2/assignment_loss‚3.9277717344688647 2loss/SRqqt1/assignment_loss‚1.6704819390165901 /loss/FRt1/detection_loss‚2.3925555694934935 0loss/SRqqt1/detection_loss‚1.377739746892263 /loss/FBt2/detection_loss‚2.1687118162711463 2loss/SRqqt3/assignment_loss‚1.4438469956882976 0loss/FBt4/assignment_loss‚0.2321897712768184 /loss/FRt3/detection_loss‚2.6101779974997044 0loss/SRqqt3/detection_loss‚1.727225892415101 /loss/FRt3/assignment_loss‚2.182500660419464 /loss/FBt1/assignment_loss‚0.510374205366033 0loss/FBt2/assignment_loss‚0.5115130462993681 1loss/SRqqt4/detection_loss‚1.0786961610226526 epoch‚9 2loss/SRqqt4/assignment_loss‚0.7783085169824029 _runtime‚9285.305835696Ò1•Ì’Ä*‚(:‡ ÃÇãÔ¥ü¯ gpu.0.gpu‚24.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚126.172' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚25.2344 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0+Í™Å*‚(:Ž  ÃÇãÔÚ™©± disk.dm-0.in‚ 420.91015625# disk.dm-0.out‚42639.37744140625 disk.sda1.out‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2266.out‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 420.91015625 cpu‚25.353938011210996 disk.rbd0.out‚ 1.0986328125 disk.drbd2265.in‚0 disk.drbd2265.out‚0$ memory_percent‚97.05815315246582* proc.memory.percent‚15.895318984985352 disk.nvme0n1.in‚ 420.91015625" proc.memory.rssMB‚ 2604.2890625* disk./.usagePercent‚15.849758742595851 disk.drbd1716.in‚0 disk.sda1.in‚0 disk.rbd1.out‚ 25.701171875 disk.sda.in‚ 0.06640625 proc.cpu.threads‚17 network.recv‚ 3.294706e+06% disk./.usageGB‚236.17285919189453 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.dm-1.out‚ 8110.83984375 disk.drbd2080.in‚0 disk.sda3.in‚ 0.06640625 disk.sda.out‚ 8110.83984375 disk.sda3.out‚ 8110.83984375( disk.nvme0c0n1.out‚42639.37744140625& disk.nvme0n1.out‚42639.37744140625' proc.memory.availableMB‚ 481.9921875 network.sent‚ 2.9676323e+07 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚ 0.0546875¹sJÛ’Æ*‚(:‡ ÒÇãÔÿÒû­ gpu.0.gpu‚39.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚115.801' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚23.1602 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚03¬H—Ç*‚(:Œ  ÒÇãÔ×°§±" disk.nvme0c0n1.in‚ 420.91015625' disk.nvme0c0n1.out‚42639.9677734375 disk.drbd2266.in‚0 disk.rbd0.in‚ 0.0546875 disk.rbd0.out‚ 1.0986328125 disk.rbd1.in‚0" proc.memory.rssMB‚ 2604.2890625 disk.nvme0n1.in‚ 420.91015625 disk.sda.out‚ 8111.27734375 disk.drbd1716.in‚0 disk.drbd2265.out‚0 disk.sda3.in‚ 0.06640625 network.recv‚ 3.299875e+06 network.sent‚ 2.9773511e+07 disk.drbd2265.in‚0 disk.rbd1.out‚ 25.7333984375# memory_percent‚97.0576524734497" disk.dm-0.out‚42639.9677734375 disk.sda1.out‚0 disk.sda.in‚ 0.06640625 disk.drbd2080.in‚0 disk.dm-0.in‚ 420.91015625* proc.memory.percent‚15.895318984985352 cpu‚25.353450070593706% disk.nvme0n1.out‚42639.9677734375 disk.sda2.in‚0 disk.drbd2080.out‚0 disk.sda1.in‚0 disk.sda3.out‚ 8111.27734375 proc.cpu.threads‚17 disk.sda2.out‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2266.out‚0( proc.memory.availableMB‚ 482.07421875% disk./.usageGB‚236.17285919189453 disk.drbd1716.out‚0 disk.dm-1.out‚ 8111.27734375* disk./.usagePercent‚15.849758742595851¾Øq2È*- _wandbruntime‚9320  _runtime‚9320ÕÈ?ÈÉ* _runtime‚9320.390026406 _step‚979 & lr-AdamW/pg1‚8.906186748124766e-08 & lr-AdamW/pg2‚8.906186748124766e-08 trainer/global_step‚11949 % _timestamp‚1.7884046957531774e+09æGØÍÊ*Ç _runtime‚9320.390026406 _step‚979 & lr-AdamW/pg1‚8.906186748124766e-08 & lr-AdamW/pg2‚8.906186748124766e-08 trainer/global_step‚11949 % _timestamp‚1.7884046957531774e+09Óʯ™;2Ë*- _wandbruntime‚9320  _runtime‚9320¡}‰òw Ì*èO OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 398837571 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 241460012 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 % _timestamp‚1.7884046957538855e+09 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 241480050 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353426938439168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16848824295936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16800276858602 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16848824295936 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157357521 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67609305 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚336981346626282 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚353426938439168 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚980 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 159458161 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 YEDeviceStatsMonSD’Ówitor.on_train_batch_start/allocated_bytes.all.allocated‚353427125442048 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16848993295872 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚353427125442048 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157357521 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 241460012 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚499 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16800441980006 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _runtime‚9320.390965918 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67608808 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚336578114143232 KEDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.current‚2 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17141800937472 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 `LDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.allocated‚320181086807040 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 398817531 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157357519 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚336578114143232 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157357519 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚497 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚336578132146176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚320181069767680 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91848854 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 241480050 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16848993295872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚373778183806976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚336981528787046 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17142017813504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91848856 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚373777942990848 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 trainer/global_step‚11949 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157357519 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚497 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 ]IDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.allocated‚336578132146176 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚320181069767680 JBDeviceStatsMonitor.on_train_batch_start/segment.small_pool.current‚199 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 YKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 216876032 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚91848854 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 UGDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.allocated‚ 241480050 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16848993295872 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚373778183806976 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 F@DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚336981528787046 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17142017813504 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚91848856 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚373777942990848 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 trainer/global_step‚11949 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚353426938439168 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16848824295936 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16800276858602 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16848824295936 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157357521 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67609305 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚336981346626282 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚353426938439168 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 240816128 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 BDeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 _step‚980 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 159458161 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚353427125442048 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16848993295872 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 VBDeviceStatsMonitor.on_train_batch_start/active_bytes.all.allocated‚353427125442048 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 157357521 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 241460012 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚499 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16800441980006 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 _runtime‚9320.390965918 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67608808 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0Ô†=Rö2Î*- _wandbruntime‚9321  _runtime‚9321fnÖ'ø&Ï*òM A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 398849977 _step‚981 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16850386447360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚336605401510912 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 241479799 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159471032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚336605401510912 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159470543 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 241499863 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚353455600942080 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16850386447360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91856231 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 398870043 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16801666023146 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚336605383507968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚356665055001600 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚373808509648384 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17143430706688 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 241479799 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16850217434112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157370178 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157370180 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 241499863 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91856233 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚337008678973162 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17143211746816 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67614312 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚320207012950016 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 398870043 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚337008861134134 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360)ò¨æIÐ*÷M ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚336605401510912 FDeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 159470543 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 241499863 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚353455600942080 QBDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.peak‚ 4541568000 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16850386447360 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚91856231 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 398870043 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16801666023146 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚336605383507968 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚356665055001600 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚373808509648384 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17143430706688 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 241479799 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16850217434112 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 157370178 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 157370180 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 241499863 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.allocated‚91856233 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚337008678973162 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17143211746816 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67614312 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚320207012950016 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 FDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 PDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 398870043 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚337008861134134 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 398849977 _step‚981 ZGDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.allocated‚16850386447360 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚336605401510912 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 241479799 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 159471032 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160ÕM`Õ2Ñ*- _wandbruntime‚9321  _runtime‚9321kˆ·î^Ò*Ø /loss/FRt3/detection_loss‚2.3386855816299263 .loss/FBt3/detection_loss‚2.893482434749603 0loss/FRt2/assignment_loss‚3.6502833832865176 1loss/FBt1/assignment_loss‚0.12163569859685847 /loss/FBt1/detection_loss‚3.1412530607647366 trainer/global_step‚11949 $ _timestamp‚1.788404696427928e+09 _step‚982 0loss/FRt1/assignment_loss‚2.9585864021067034 0loss/FRt3/assignment_loss‚2.5176469168879767 2loss/SRqqt1/assignment_loss‚1.4398330577438878 1loss/FBt2/assignment_loss‚0.37697550738936186 .loss/FRt4/detection_loss‚2.008148898681005 epoch‚9 2loss/SRqqt4/assignment_loss‚0.9701196862983384 /loss/FBt4/detection_loss‚0.7862814882552872 1loss/SRqqt2/detection_loss‚2.4321395425960937 1loss/SRqqt4/detection_loss‚1.4224192123594028 .loss/FBt2/detection_loss‚3.076509340242906 2loss/SRqqt2/assignment_loss‚2.2381469155180045 2loss/SRqqt3/assignment_loss‚1.4440708572843246 /loss/FRt1/detection_loss‚1.8991592918571674 _runtime‚9321.064663277 1loss/FBt3/assignment_loss‚0.19923871061763448 1loss/SRqqt1/detection_loss‚1.6717784416771704 0loss/SRqqt3/detection_loss‚1.582375768572092 %loss/total_loss‚44.59854183080412 0loss/FRt4/assignment_loss‚2.0012541569414593 0loss/FBt4/assignment_loss‚0.2109316332053216 /loss/FRt2/detection_loss‚3.2175858435423477Ëí,„cÓ*Ý 0loss/SRqqt3/detection_loss‚1.582375768572092 %loss/total_loss‚44.59854183080412 0loss/FRt4/assignment_loss‚2.0012541569414593 0loss/FBt4/assignment_loss‚0.2109316332053216 /loss/FRt2/detection_loss‚3.2175858435423477 /loss/FRt3/detection_loss‚2.3386855816299263 .loss/FBt3/detection_loss‚2.893482434749603 0loss/FRt2/assignment_loss‚3.6502833832865176 1loss/FBt1/assignment_loss‚0.12163569859685847 /loss/FBt1/detection_loss‚3.1412530607647366 trainer/global_step‚11949 $ _timestamp‚1.788404696427928e+09 _step‚982 0loss/FRt1/assignment_loss‚2.9585864021067034 0loss/FRt3/assignment_loss‚2.5176469168879767 2loss/SRqqt1/assignment_loss‚1.4398330577438878 1loss/FBt2/assignment_loss‚0.37697550738936186 .loss/FRt4/detection_loss‚2.008148898681005 epoch‚9 2loss/SRqqt4/assignment_loss‚0.9701196862983384 /loss/FBt4/detection_loss‚0.7862814882552872 1loss/SRqqt2/detection_loss‚2.4321395425960937 1loss/SRqqt4/detection_loss‚1.4224192123594028 .loss/FBt2/detection_loss‚3.076509340242906 2loss/SRqqt2/assignment_loss‚2.2381469155180045 2loss/SRqqt3/assignment_loss‚1.4440708572843246 /loss/FRt1/detection_loss‚1.8991592918571674 _runtime‚9321.064663277 1loss/FBt3/assignment_loss‚0.19923871061763448 1loss/SRqqt1/detection_loss‚1.6717784416771704Ön¥/pÔ*‚(:’ áÇãÔ´ý•® gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚135.089' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.017799999999998 gpu.0.smClock‚1365 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ÌTK—Õ*‚(:Œ  áÇãÔ€ËÕ° disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.rbd1.out‚ 25.9130859375 disk.nvme0n1.in‚ 420.91015625 cpu‚25.352940314992544 network.sent‚ 2.9779438e+07 disk.dm-1.out‚ 8111.89453125# memory_percent‚97.0595121383667 network.recv‚ 3.304022e+06 disk.sda1.out‚0 disk.sda3.in‚ 0.06640625 disk.rbd1.in‚0 disk.sda.out‚ 8111.89453125* proc.memory.percent‚15.895318984985352* disk./.usagePercent‚15.849767446852017 disk.sda2.out‚0 disk.rbd0.in‚ 0.0546875 disk.sda.in‚ 0.06640625 proc.cpu.threads‚17 disk.dm-1.in‚ 0.06640625 disk.drbd2266.in‚0 disk.dm-0.in‚ 420.91015625 disk.sda3.out‚ 8111.89453125 disk.drbd1716.in‚0 disk.sda1.in‚0( proc.memory.availableMB‚ 481.76953125% disk./.usageGB‚236.17298889160156 disk.drbd2265.out‚0 disk.drbd2080.in‚0 disk.rbd0.out‚ 1.0986328125" disk.dm-0.out‚42640.2607421875" disk.nvme0c0n1.in‚ 420.91015625% disk.nvme0n1.out‚42640.2607421875" proc.memory.rssMB‚ 2604.2890625 disk.sda2.in‚0 disk.drbd1716.out‚0 disk.drbd2080.out‚0' disk.nvme0c0n1.out‚42640.2607421875ȳ£yÖ*‚(:’ ðÇãÔ¹Ú¡® gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚138.866' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚27.773200000000003 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0 À ‚‘×*‚(:†  ðÇãÔÄöï°$ disk.nvme0n1.out‚42641.201171875 disk.sda.in‚ 0.06640625 disk.rbd0.in‚ 0.0546875 disk.sda3.in‚ 0.06640625' proc.memory.availableMB‚ 481.7734375) disk./.usagePercent‚15.84974773427188 disk.sda.out‚ 8112.82421875 disk.drbd1716.out‚0 disk.drbd2265.out‚0 disk.rbd0.out‚ 1.0986328125* proc.memory.percent‚15.895318984985352 cpu‚25.35248086576019 network.recv‚ 3.310388e+06 disk.dm-1.out‚ 8112.82421875 disk.rbd1.in‚0& disk.nvme0c0n1.out‚42641.201171875$ disk./.usageGB‚236.1726951599121 disk.drbd2266.in‚0" disk.nvme0c0n1.in‚ 420.91015625 disk.dm-0.in‚ 420.91015625! disk.dm-0.out‚42641.201171875 disk.rbd1.out‚ 25.9755859375 disk.nvme0n1.in‚ 420.91015625" proc.memory.rssMB‚ 2604.2890625 disk.sda2.in‚0 disk.drbd2080.in‚0$ memory_percent‚97.05948829650879 disk.drbd1716.in‚0 disk.sda1.in‚0 disk.sda1.out‚0 proc.cpu.threads‚17 network.sent‚ 2.9888275e+07 disk.sda2.out‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2266.out‚0 disk.drbd2265.in‚0 disk.drbd2080.out‚0 disk.sda3.out‚ 8112.82421875¼,Y¢2Ø*- _wandbruntime‚9356  _runtime‚9356‘¥u–ÉÙ*à % _timestamp‚1.7884047314364533e+09 _runtime‚ 9356.07329001 _step‚983 ' lr-AdamW/pg1‚3.7099781662185795e-08 ' lr-AdamW/pg2‚3.7099781662185795e-08 trainer/global_step‚11999|þ/pÎÚ*È _step‚983 ' lr-AdamW/pg1‚3.7099781662185795e-08 ' lr-AdamW/pg2‚3.7099781662185795e-08 trainer/global_step‚11999 % _timestamp‚1.7884047314364533e+09 _runtime‚ 9356.07329001×È[Àè2Û*- _wandbruntime‚9356  _runtime‚9356¨w¹Ÿî'Ü*èO JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚354860077998592 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92217704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 242472752 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 trainer/global_step‚11999 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚375294678902272 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 400463223 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚338347977561490 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚354860265001472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 160094958 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 400443183 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16869913798414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 242452714 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67877252 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16869748677010 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚92217706 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚321478228884480 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157990469 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚337941582380032 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16918664618496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16918664618496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67876722 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 % _timestamp‚1.7884047314371405e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16918495618560 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 242452714 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157990469 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚358081813404672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚358081789464576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157990471 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16918495618560 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚337941582380032 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248Œª ‡ó'Ý*íO G>DeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 400463223 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚338347977561490 XJDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.current‚ 165121404 MFDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.peak‚26 YEDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.allocated‚354860265001472 E;DeviceStatsMonitor.on_train_batch_start/allocation.all.peak‚24401 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 @:DeviceStatsMonitor.on_train_batch_start/num_oom_rejections‚0 GADeviceStatsMonitor.on_train_batch_start/oversize_allocations.peak‚0 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 160094958 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 C;DeviceStatsMonitor.on_train_batch_start/segment.all.current‚416 LDDeviceStatsMonitor.on_train_batch_start/segment.small_pool.allocated‚199 F8DeviceStatsMonitor.on_train_batch_start/active.all.freed‚ 400443183 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 _LDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.allocated‚16869913798414 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 O@DeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 242452714 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.allocated‚67877252 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16869748677010 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚92217706 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚321478228884480 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 157990469 YEDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.freed‚337941582380032 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16918664618496 UGDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.current‚ 168999936 60DeviceStatsMonitor.on_train_batch_start/num_ooms‚0 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16918664618496 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.freed‚67876722 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 % _timestamp‚1.7884047314371405e+09 [HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.freed‚16918495618560 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 A7DeviceStatsMonitor.on_train_batch_start/active.all.peak‚24401 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 242452714 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.freed‚ 157990469 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 VGDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.peak‚ 4390226944 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚358081813404672 aMDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.freed‚358081789464576 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 HBDeviceStatsMonitor.on_train_batch_start/oversize_allocations.freed‚0 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 157990471 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16918495618560 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 SDDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.peak‚ 4541568000 QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚530 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 232427520 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 TGDeviceStatsMonitor.on_train_batch_start/active_bytes.large_pool.current‚18002944 \HDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.freed‚337941582380032 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 WIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.current‚ 417333248 JDeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚354860077998592 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 TGDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.freed‚92217704 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 TFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.peak‚ 417333248 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 242472752 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚532 trainer/global_step‚11999 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚375294678902272 BDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 _runtime‚9356.750489124 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 400475629 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚337968869747712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17214283994112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400495695 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚321504172066816 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 _step‚985 WIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 >6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92225081 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158003128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚354888740501504 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160107098 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16871137841554 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67882540 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67882017 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 242472501 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788404732113215e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16919888756736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17214069228544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚338375492069342 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158003130 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 trainer/global_step‚11999 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 400495695 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16920057769984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚337968869747712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚338375309908370 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚375325010347008 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 242492565 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚354888927517696 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 400475629 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚358110726352896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158003130 P6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92225081 ;4DeviceStatsMonitor.on_train_batch_end/max_split_size‚-1 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 [MDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.current‚ 214765568 >8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158003128 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚354888740501504 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160107098 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16871137841554 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚67882540 REDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.freed‚67882017 E?DeviceStatsMonitor.on_train_batch_end/oversize_allocations.peak‚0 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 242472501 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 REDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.current‚18002944 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.freed‚0 $ _timestamp‚1.788404732113215e+09 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16919888756736 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17214069228544 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 J;DeviceStatsMonitor.on_train_batch_end/active_bytes.all.peak‚ 4912020992 MGDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.current‚2 @:DeviceStatsMonitor.on_train_batch_end/num_sync_all_streams‚2 WCDeviceStatsMonitor.on_train_batch_end/requested_bytes.all.allocated‚338375492069342 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158003130 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 238705664 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 UHDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.current‚17039360 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 trainer/global_step‚11999 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 400495695 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16920057769984 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚337968869747712 ;5DeviceStatsMonitor.on_train_batch_end/num_device_free‚0 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚338375309908370 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 H@DeviceStatsMonitor.on_train_batch_end/segment.small_pool.current‚199 \HDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.allocated‚375325010347008 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 242492565 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚354888927517696 H:DeviceStatsMonitor.on_train_batch_end/allocation.all.freed‚ 400475629 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚358110726352896 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158003130 PDeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 _runtime‚9356.750489124 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 400475629 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚337968869747712 PBDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.peak‚ 410712064 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17214283994112 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_end/segment.large_pool.allocated‚217 FDeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 400495695 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 QBDeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.allocated‚ 5041553408 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚321504172066816 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 _step‚985Ùw Wþ2á*- _wandbruntime‚9356  _runtime‚9356*q[â*Õ 1loss/SRqqt2/assignment_loss‚2.029572869566354 2loss/SRqqt3/assignment_loss‚1.5210551098691967 /loss/FRt1/detection_loss‚2.2592204377055167 /loss/FRt4/detection_loss‚1.6912974766283124 0loss/SRqqt2/detection_loss‚2.464832078326832 /loss/FBt2/detection_loss‚2.9362513720989227 /loss/FBt3/detection_loss‚3.0585310220718385 %loss/total_loss‚44.15282464522813 1loss/FBt2/assignment_loss‚0.20616015753820421 0loss/SRqqt4/detection_loss‚1.002710257191211 /loss/FBt1/detection_loss‚2.8803360958894095 /loss/FRt3/assignment_loss‚3.278375401308662 2loss/FBt1/assignment_loss‚0.014129145497709791 1loss/FBt4/assignment_loss‚0.09645667957893522 .loss/FRt2/detection_loss‚2.923892789416843 /loss/FRt3/detection_loss‚2.5101267930708433 /loss/FBt4/detection_loss‚0.8356368568105003 _step‚986 0loss/FRt4/assignment_loss‚1.5598475381180092 1loss/SRqqt1/assignment_loss‚1.562310121792297 /loss/FBt3/assignment_loss‚0.155847376022939 _runtime‚9356.751558722 /loss/FRt2/assignment_loss‚3.467117596555639 1loss/SRqqt1/detection_loss‚1.7405303599169621 epoch‚9 % _timestamp‚1.7884047321148381e+09 /loss/FRt1/assignment_loss‚3.166310634613037 2loss/SRqqt4/assignment_loss‚1.1712865544007702 1loss/SRqqt3/detection_loss‚1.6209899212391872 trainer/global_step‚11999=ÞE`ã*Ú 1loss/FBt2/assignment_loss‚0.20616015753820421 0loss/SRqqt4/detection_loss‚1.002710257191211 /loss/FBt1/detection_loss‚2.8803360958894095 /loss/FRt3/assignment_loss‚3.278375401308662 2loss/FBt1/assignment_loss‚0.014129145497709791 1loss/FBt4/assignment_loss‚0.09645667957893522 .loss/FRt2/detection_loss‚2.923892789416843 /loss/FRt3/detection_loss‚2.5101267930708433 /loss/FBt4/detection_loss‚0.8356368568105003 _step‚986 0loss/FRt4/assignment_loss‚1.5598475381180092 1loss/SRqqt1/assignment_loss‚1.562310121792297 /loss/FBt3/assignment_loss‚0.155847376022939 _runtime‚9356.751558722 /loss/FRt2/assignment_loss‚3.467117596555639 1loss/SRqqt1/detection_loss‚1.7405303599169621 epoch‚9 % _timestamp‚1.7884047321148381e+09 /loss/FRt1/assignment_loss‚3.166310634613037 2loss/SRqqt4/assignment_loss‚1.1712865544007702 1loss/SRqqt3/detection_loss‚1.6209899212391872 trainer/global_step‚11999 1loss/SRqqt2/assignment_loss‚2.029572869566354 2loss/SRqqt3/assignment_loss‚1.5210551098691967 /loss/FRt1/detection_loss‚2.2592204377055167 /loss/FRt4/detection_loss‚1.6912974766283124 0loss/SRqqt2/detection_loss‚2.464832078326832 /loss/FBt2/detection_loss‚2.9362513720989227 /loss/FBt3/detection_loss‚3.0585310220718385 %loss/total_loss‚44.15282464522813ÚBÅm?’ä*‚(:‡ ÿÇãÔô‚® gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚144.824' gpu.0.enforcedPowerLimitWatts‚500.0 gpu.0.powerPercent‚28.9648 gpu.0.smClock‚1380 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0¶ÏÞµ•å*‚(:Š  ÿÇãÔªÂò° disk.drbd2080.in‚0 disk.rbd0.out‚ 1.0986328125 disk.sda3.in‚ 0.06640625 disk.dm-1.out‚ 8113.21484375( proc.memory.availableMB‚ 481.55078125* disk./.usagePercent‚15.849751830392428 disk.rbd1.in‚0 disk.dm-1.in‚ 0.06640625 network.recv‚ 3.315153e+06 disk.drbd2080.out‚0 disk.sda2.in‚0$ memory_percent‚97.06084728240967 disk.sda1.in‚0" disk.nvme0c0n1.in‚ 420.91015625 disk.nvme0n1.in‚ 420.91015625! disk.dm-0.out‚42641.443359375 disk.rbd0.in‚ 0.0546875 disk.dm-0.in‚ 420.91015625 disk.rbd1.out‚ 25.9755859375 disk.sda.in‚ 0.06640625 disk.drbd1716.out‚0* proc.memory.percent‚15.895318984985352 proc.cpu.threads‚17 network.sent‚ 2.9894596e+07 disk.drbd2265.out‚0 disk.sda3.out‚ 8113.21484375 disk.drbd1716.in‚0% disk./.usageGB‚236.17275619506836 disk.drbd2265.in‚0& disk.nvme0c0n1.out‚42641.443359375$ disk.nvme0n1.out‚42641.443359375 disk.sda2.out‚0 disk.drbd2266.in‚0 disk.drbd2266.out‚0" proc.memory.rssMB‚ 2604.2890625 disk.sda1.out‚0 disk.sda.out‚ 8113.21484375 cpu‚25.351970668074124Y‘$æ*‚(:’ ŽÈãÔËÔƒ® gpu.0.gpu‚32.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚33.0 gpu.0.powerWatts‚129.949' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚25.989800000000002 gpu.0.smClock‚1320 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0!Pv8–ç*‚(:‹  ŽÈãԎݱ disk.sda3.out‚ 8113.76953125 disk.dm-1.out‚ 8113.76953125 disk.drbd2266.in‚0* disk./.usagePercent‚15.849750550354758 disk.drbd2080.in‚0( disk.nvme0c0n1.out‚42642.24365234375 proc.cpu.threads‚17 disk.dm-1.in‚ 0.06640625 disk.drbd2265.out‚0# disk.dm-0.out‚42642.24365234375 disk.sda1.in‚0$ memory_percent‚97.05972671508789* proc.memory.percent‚15.895318984985352 disk.drbd1716.in‚0& proc.memory.availableMB‚ 481.734375 disk.drbd1716.out‚0 disk.drbd2265.in‚0& disk.nvme0n1.out‚42642.24365234375" proc.memory.rssMB‚ 2604.2890625 disk.rbd0.out‚ 1.0986328125 disk.sda1.out‚0 disk.sda3.in‚ 0.06640625 disk.rbd1.out‚ 26.0078125 network.sent‚ 3.0002489e+07 network.recv‚ 3.320823e+06 disk.sda.in‚ 0.06640625 disk.sda2.in‚0 disk.sda.out‚ 8113.76953125 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 420.91015625 disk.nvme0n1.in‚ 420.91015625 cpu‚25.351485326517874% disk./.usageGB‚236.17273712158203 disk.sda2.out‚0 disk.drbd2080.out‚0 disk.rbd0.in‚ 0.0546875 disk.drbd2266.out‚0 disk.dm-0.in‚ 420.91015625­#lªè*‚(:’ ÈãÔñ†·® gpu.0.gpu‚23.0 gpu.0.memory‚2, gpu.0.memoryAllocated‚6.5064239501953125) gpu.0.memoryAllocatedBytes‚ 5588975616 gpu.0.temp‚34.0 gpu.0.powerWatts‚147.246' gpu.0.enforcedPowerLimitWatts‚500.0) gpu.0.powerPercent‚29.449200000000005 gpu.0.smClock‚1395 gpu.0.memoryClock‚1593! gpu.0.correctedMemoryErrors‚0# gpu.0.uncorrectedMemoryErrors‚0ù„·z”é*‚(:‰  ÈãÔ妞° disk.rbd1.out‚ 26.1015625 proc.cpu.threads‚17 disk.sda.in‚ 0.06640625 disk.sda.out‚ 8114.2734375 disk.sda2.out‚0 disk.dm-1.out‚ 8114.2734375 disk.drbd2265.out‚0 disk.sda1.in‚0 disk.drbd1716.out‚0 disk.drbd2080.in‚0 disk.drbd2080.out‚0 disk.rbd0.in‚ 0.0546875# disk.dm-0.out‚42642.47021484375$ memory_percent‚97.06122875213623( proc.memory.availableMB‚ 481.48828125 disk.drbd2266.out‚0 network.sent‚ 3.0008541e+07% disk./.usageGB‚236.17277145385742 disk.nvme0n1.in‚ 420.91015625 disk.sda2.in‚0 disk.drbd1716.in‚0 disk.dm-1.in‚ 0.06640625 disk.drbd2266.in‚0 disk.rbd1.in‚0" disk.nvme0c0n1.in‚ 420.91015625( disk.nvme0c0n1.out‚42642.47021484375 disk.drbd2265.in‚0 disk.rbd0.out‚ 1.0986328125* disk./.usagePercent‚15.849752854422567 disk.sda3.out‚ 8114.2734375& disk.nvme0n1.out‚42642.47021484375" proc.memory.rssMB‚ 2604.2890625 cpu‚25.351012151167833 network.recv‚ 3.32536e+06 disk.sda1.out‚0* proc.memory.percent‚15.895318984985352 disk.sda3.in‚ 0.06640625 disk.dm-0.in‚ 420.91015625ìg·ñ2ê*- _wandbruntime‚9391  _runtime‚9391”DuàÇë*Á _step‚987 & lr-AdamW/pg1‚7.531408330734621e-09 & lr-AdamW/pg2‚7.531408330734621e-09 trainer/global_step‚12049 $ _timestamp‚1.788404767177982e+09 _runtime‚9391.8148426326k&Ìì*Æ trainer/global_step‚12049 $ _timestamp‚1.788404767177982e+09 _runtime‚9391.814842632 _step‚987 & lr-AdamW/pg1‚7.531408330734621e-09 & lr-AdamW/pg2‚7.531408330734621e-09Û¦Ýë2í*- _wandbruntime‚9391  _runtime‚9391Ã7Qħî*èO QIDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.current‚513 UADeviceStatsMonitor.on_train_batch_start/requested_bytes.all.freed‚339714608496698 M?DeviceStatsMonitor.on_train_batch_start/active.small_pool.freed‚ 243445416 UGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.peak‚ 410712064 JBDeviceStatsMonitor.on_train_batch_start/inactive_split.all.current‚515 O@DeviceStatsMonitor.on_train_batch_start/requested_bytes.all.peak‚ 4757245257 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.freed‚0 trainer/global_step‚12049 M?DeviceStatsMonitor.on_train_batch_start/active.large_pool.freed‚ 158623419 \IDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.allocated‚16988335941120 @8DeviceStatsMonitor.on_train_batch_start/segment.all.peak‚416 _step‚988 L=DeviceStatsMonitor.on_train_batch_start/active_bytes.all.peak‚ 4912020992 QCDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.freed‚ 243445416 [HDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.freed‚16939220495418 UGDeviceStatsMonitor.on_train_batch_start/requested_bytes.small_pool.peak‚ 406789969 QCDeviceStatsMonitor.on_train_batch_start/active.large_pool.allocated‚ 158623421 =7DeviceStatsMonitor.on_train_batch_start/num_device_free‚0 ICDeviceStatsMonitor.on_train_batch_start/oversize_segments.allocated‚0 XIDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.current‚ 4624220160 % _timestamp‚1.7884047671786783e+09 UADeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.freed‚356293217558016 NFDeviceStatsMonitor.on_train_batch_start/inactive_split.small_pool.peak‚985 \ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.current‚23940096 =6DeviceStatsMonitor.on_train_batch_start/max_split_size‚-1 MGDeviceStatsMonitor.on_train_batch_start/reserved_bytes.small_pool.freed‚0 ]ODeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.current‚ 218973184 ?9DeviceStatsMonitor.on_train_batch_start/num_alloc_retries‚0 _runtime‚9391.815717603 H>DeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚359527460815872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 160733887 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚376810948344832 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 402088875 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚92586556 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚376811191258112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚339714790657462 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚322775388001280 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16988335941120 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17283730442240 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 243465454 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 160734402 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17283511469056 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 SZ/¹GEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16988166941184 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158623421 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚356293217558016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 JDeviceStatsMonitor.on_train_batch_start/active.small_pool.peak‚24128 VHDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.current‚ 242913280 eQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.allocated‚359527460815872 LDDeviceStatsMonitor.on_train_batch_start/segment.large_pool.allocated‚217 N@DeviceStatsMonitor.on_train_batch_start/inactive_split.all.freed‚ 160733887 JDDeviceStatsMonitor.on_train_batch_start/oversize_allocations.current‚0 SDDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.allocated‚ 5041553408 F@DeviceStatsMonitor.on_train_batch_start/segment.large_pool.freed‚0 G?DeviceStatsMonitor.on_train_batch_start/segment.small_pool.peak‚199 GADeviceStatsMonitor.on_train_batch_start/active.large_pool.current‚2 ZFDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.freed‚376810948344832 @8DeviceStatsMonitor.on_train_batch_start/num_device_alloc‚416 LFDeviceStatsMonitor.on_train_batch_start/oversize_allocations.allocated‚0 E?DeviceStatsMonitor.on_train_batch_start/oversize_segments.freed‚0 D:DeviceStatsMonitor.on_train_batch_start/active.all.current‚20040 N@DeviceStatsMonitor.on_train_batch_start/allocation.all.allocated‚ 402088875 XKDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.allocated‚92586556 ^JDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.allocated‚376811191258112 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 YEDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.allocated‚339714790657462 \HDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.freed‚322775388001280 _LDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.allocated‚16988335941120 VGDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.peak‚ 4541568000 XJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.small_pool.current‚ 168999936 OIDeviceStatsMonitor.on_train_batch_start/inactive_split.large_pool.current‚2 dQDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.allocated‚17283730442240 GADeviceStatsMonitor.on_train_batch_start/oversize_segments.current‚0 F@DeviceStatsMonitor.on_train_batch_start/segment.small_pool.freed‚0 QCDeviceStatsMonitor.on_train_batch_start/active.small_pool.allocated‚ 243465454 QCDeviceStatsMonitor.on_train_batch_start/allocated_bytes.all.current‚ 187002880 KBDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.peak‚2460 LBDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.peak‚24128 RDDeviceStatsMonitor.on_train_batch_start/inactive_split.all.allocated‚ 160734402 `MDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.small_pool.freed‚17283511469056 N?DeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.peak‚ 5041553408 H?DeviceStatsMonitor.on_train_batch_start/inactive_split.all.peak‚1006 D>DeviceStatsMonitor.on_train_batch_start/oversize_segments.peak‚0 UFDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.peak‚ 4624220160 SEDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.all.peak‚ 269877760 ZLDeviceStatsMonitor.on_train_batch_start/inactive_split_bytes.large_pool.peak‚ 104436224 ?9DeviceStatsMonitor.on_train_batch_start/segment.all.freed‚0 JBDeviceStatsMonitor.on_train_batch_start/segment.large_pool.current‚217 XEDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.freed‚16988166941184 RDDeviceStatsMonitor.on_train_batch_start/active_bytes.small_pool.peak‚ 410712064 H>DeviceStatsMonitor.on_train_batch_start/allocation.all.current‚20040 E=DeviceStatsMonitor.on_train_batch_start/segment.all.allocated‚416 UGDeviceStatsMonitor.on_train_batch_start/allocation.large_pool.allocated‚ 158623421 BDeviceStatsMonitor.on_train_batch_start/active.large_pool.peak‚2460 N@DeviceStatsMonitor.on_train_batch_start/active_bytes.all.current‚ 187002880 OEDeviceStatsMonitor.on_train_batch_start/allocation.small_pool.current‚20038 QCDeviceStatsMonitor.on_train_batch_start/requested_bytes.all.current‚ 182160764 WJDeviceStatsMonitor.on_train_batch_start/requested_bytes.large_pool.current‚17039360 QBDeviceStatsMonitor.on_train_batch_start/reserved_bytes.all.current‚ 5041553408 ZKDeviceStatsMonitor.on_train_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 R>DeviceStatsMonitor.on_train_batch_start/active_bytes.all.freed‚356293217558016 WJDeviceStatsMonitor.on_train_batch_start/allocated_bytes.large_pool.current‚18002944 G?DeviceStatsMonitor.on_train_batch_start/segment.large_pool.peak‚217 KADeviceStatsMonitor.on_train_batch_start/active.small_pool.current‚20038 J6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 243485267 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92593931 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16940774781574 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚339332337984512 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68153327 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚359556349824000 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158636078 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 _step‚989 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17284930693632 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚9392.483722011 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚515 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚356321880060928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160746745 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚12049 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158636080 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402121347 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16989560079360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚356322067077120 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚339332319981568 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚356322067077120 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16989560079360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16989729092608 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 % _timestamp‚1.7884047678463542e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 402101281 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 160747260 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17285139167744 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16940609659962 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚339741940843578 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚322801348222976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158636078 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚339332337984512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚359556373764096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 243485267 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚322801331183616 PDeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚339332319981568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401ùŒïYyò*÷M XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.peak‚ 239880192 D>DeviceStatsMonitor.on_train_batch_end/segment.large_pool.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.freed‚339332319981568 C9DeviceStatsMonitor.on_train_batch_end/allocation.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.small_pool.freed‚ 243465203 ICDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/allocation.small_pool.freed‚ 243465203 OGDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.current‚513 B8DeviceStatsMonitor.on_train_batch_end/active.all.current‚20066 H:DeviceStatsMonitor.on_train_batch_end/active.all.allocated‚ 402121347 OADeviceStatsMonitor.on_train_batch_end/active.large_pool.allocated‚ 158636080 I?DeviceStatsMonitor.on_train_batch_end/active.small_pool.current‚20064 >6DeviceStatsMonitor.on_train_batch_end/num_device_alloc‚416 SEDeviceStatsMonitor.on_train_batch_end/allocation.small_pool.allocated‚ 243485267 F=DeviceStatsMonitor.on_train_batch_end/inactive_split.all.peak‚1006 REDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.freed‚92593931 ]JDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.allocated‚16940774781574 O@DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.current‚ 5041553408 ^JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.allocated‚339332337984512 KDDeviceStatsMonitor.on_train_batch_end/inactive_split.large_pool.peak‚26 =7DeviceStatsMonitor.on_train_batch_end/segment.all.freed‚0 SEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.peak‚ 410712064 VIDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.allocated‚68153327 _KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.freed‚359556349824000 M>DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.peak‚ 4757245257 TEDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.peak‚ 4390226944 OADeviceStatsMonitor.on_train_batch_end/allocation.large_pool.freed‚ 158636078 VHDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.current‚ 165121612 _step‚989 TEDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 TFDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.current‚ 232414208 ^KDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.freed‚17284930693632 SEDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.peak‚ 406789969 XIDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 E8DeviceStatsMonitor.on_train_batch_end/num_oom_rejections‚0 _runtime‚9392.483722011 VHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.current‚ 169013248 H@DeviceStatsMonitor.on_train_batch_end/inactive_split.all.current‚515 S?DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.freed‚356321880060928 UHDeviceStatsMonitor.on_train_batch_end/allocated_bytes.large_pool.current‚18002944 L>DeviceStatsMonitor.on_train_batch_end/inactive_split.all.freed‚ 160746745 GADeviceStatsMonitor.on_train_batch_end/oversize_segments.allocated‚0 SDDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.peak‚ 4624220160 trainer/global_step‚12049 L>DeviceStatsMonitor.on_train_batch_end/active_bytes.all.current‚ 187016192 SEDeviceStatsMonitor.on_train_batch_end/allocation.large_pool.allocated‚ 158636080 QCDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.all.peak‚ 269877760 VGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.current‚ 4624220160 L>DeviceStatsMonitor.on_train_batch_end/allocation.all.allocated‚ 402121347 VCDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.freed‚16989560079360 E?DeviceStatsMonitor.on_train_batch_end/oversize_segments.current‚0 C;DeviceStatsMonitor.on_train_batch_end/segment.all.allocated‚416 H@DeviceStatsMonitor.on_train_batch_end/segment.large_pool.current‚217 D>DeviceStatsMonitor.on_train_batch_end/segment.small_pool.freed‚0 T@DeviceStatsMonitor.on_train_batch_end/active_bytes.all.allocated‚356322067077120 WCDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.freed‚339332319981568 WCDeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.allocated‚356322067077120 YFDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.freed‚16989560079360 HBDeviceStatsMonitor.on_train_batch_end/oversize_allocations.current‚0 UGDeviceStatsMonitor.on_train_batch_end/reserved_bytes.small_pool.current‚ 417333248 ]JDeviceStatsMonitor.on_train_batch_end/allocated_bytes.small_pool.allocated‚16989729092608 LDDeviceStatsMonitor.on_train_batch_end/inactive_split.small_pool.peak‚985 E=DeviceStatsMonitor.on_train_batch_end/segment.large_pool.peak‚217 % _timestamp‚1.7884047678463542e+09 D6DeviceStatsMonitor.on_train_batch_end/active.all.freed‚ 402101281 OADeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.current‚ 187016192 PBDeviceStatsMonitor.on_train_batch_end/inactive_split.all.allocated‚ 160747260 bODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.small_pool.allocated‚17285139167744 F@DeviceStatsMonitor.on_train_batch_end/oversize_allocations.freed‚0 YFDeviceStatsMonitor.on_train_batch_end/requested_bytes.small_pool.freed‚16940609659962 KEDeviceStatsMonitor.on_train_batch_end/reserved_bytes.large_pool.freed‚0 F6DeviceStatsMonitor.on_train_batch_end/segment.all.peak‚416 E=DeviceStatsMonitor.on_train_batch_end/segment.small_pool.peak‚199 SEDeviceStatsMonitor.on_train_batch_end/active_bytes.small_pool.current‚ 169013248 M>DeviceStatsMonitor.on_train_batch_end/allocated_bytes.all.peak‚ 4912020992 ZMDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.current‚23940096 S?DeviceStatsMonitor.on_train_batch_end/requested_bytes.all.freed‚339741940843578 ^JDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.allocated‚322801348222976 ?5DeviceStatsMonitor.on_train_batch_end/active.all.peak‚24401 K=DeviceStatsMonitor.on_train_batch_end/active.large_pool.freed‚ 158636078 [GDeviceStatsMonitor.on_train_batch_end/active_bytes.large_pool.allocated‚339332337984512 J@DeviceStatsMonitor.on_train_batch_end/allocation.small_pool.peak‚24128 cODeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.allocated‚359556373764096 A9DeviceStatsMonitor.on_train_batch_end/segment.all.current‚416 JBDeviceStatsMonitor.on_train_batch_end/segment.small_pool.allocated‚199 =7DeviceStatsMonitor.on_train_batch_end/num_alloc_retries‚0 JDDeviceStatsMonitor.on_train_batch_end/oversize_allocations.allocated‚0 OADeviceStatsMonitor.on_train_batch_end/requested_bytes.all.current‚ 182160972 L=DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.peak‚ 5041553408 E?DeviceStatsMonitor.on_train_batch_end/active.large_pool.current‚2 OADeviceStatsMonitor.on_train_batch_end/active.small_pool.allocated‚ 243485267 4.DeviceStatsMonitor.on_train_batch_end/num_ooms‚0 D>DeviceStatsMonitor.on_train_batch_end/reserved_bytes.all.freed‚0 I@DeviceStatsMonitor.on_train_batch_end/allocation.large_pool.peak‚2460 XJDeviceStatsMonitor.on_train_batch_end/inactive_split_bytes.large_pool.peak‚ 104436224 C=DeviceStatsMonitor.on_train_batch_end/oversize_segments.freed‚0 ZFDeviceStatsMonitor.on_train_batch_end/requested_bytes.large_pool.freed‚322801331183616 PDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚416 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚358409947174400 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 145311232 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 244624492 _step‚991 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚274 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚985 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚17359035204608 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 B;DeviceStatsMonitor.on_validation_batch_start/max_split_size‚-1 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚324716251645952 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 417333248 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚199 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 4912020992 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 161818933 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1006 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 410712064 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚276 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚17013274584906 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚341347506677760 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 4541568000 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2460 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚26 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚361673442465792 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 103378432 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚416 ]PDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.allocated‚68611420 YLDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.freed‚68611146 cODeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.all.allocated‚379032501610496 trainer/global_step‚12089 RDDeviceStatsMonitor.on_validation_batch_start/active.large_pool.freed‚ 159667326 LCDeviceStatsMonitor.on_validation_batch_start/active.large_pool.peak‚2460 RDDeviceStatsMonitor.on_validation_batch_start/active.small_pool.freed‚ 244624492 `MDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.small_pool.freed‚17062458499584 ^PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.allocated‚ 417333248 aMDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.freed‚341347488674816 BDeviceStatsMonitor.on_validation_batch_start/num_alloc_retries‚0 E=DeviceStatsMonitor.on_validation_batch_start/segment.all.peak‚416 MCDeviceStatsMonitor.on_validation_batch_start/active.small_pool.peak‚24128 WCDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.freed‚358409947174400 VHDeviceStatsMonitor.on_validation_batch_start/allocated_bytes.all.current‚ 145311232 \ODeviceStatsMonitor.on_validation_batch_start/allocated_bytes.large_pool.current‚18002944 VHDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.freed‚ 244624492 _step‚991 QKDeviceStatsMonitor.on_validation_batch_start/oversize_allocations.allocated‚0 VNDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.current‚274 SKDeviceStatsMonitor.on_validation_batch_start/inactive_split.small_pool.peak‚985 iVDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.allocated‚17359035204608 _QDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.peak‚ 239880192 B;DeviceStatsMonitor.on_validat *˜`ion_batch_start/max_split_size‚-1 eQDeviceStatsMonitor.on_validation_batch_start/requested_bytes.large_pool.allocated‚324716251645952 _PDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.large_pool.allocated‚ 4624220160 YKDeviceStatsMonitor.on_validation_batch_start/reserved_bytes.small_pool.peak‚ 417333248 FDeviceStatsMonitor.on_validation_batch_start/segment.all.freed‚0 LDDeviceStatsMonitor.on_validation_batch_start/segment.small_pool.peak‚199 QBDeviceStatsMonitor.on_validation_batch_start/active_bytes.all.peak‚ 4912020992 TJDeviceStatsMonitor.on_validation_batch_start/allocation.small_pool.current‚15977 WIDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.allocated‚ 161818933 MDDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.peak‚1006 aTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.current‚23940096 PFDeviceStatsMonitor.on_validation_batch_start/active.small_pool.current‚15977 WIDeviceStatsMonitor.on_validation_batch_start/active_bytes.small_pool.peak‚ 410712064 MCDeviceStatsMonitor.on_validation_batch_start/allocation.all.current‚15979 OGDeviceStatsMonitor.on_validation_batch_start/inactive_split.all.current‚276 `MDeviceStatsMonitor.on_validation_batch_start/requested_bytes.small_pool.freed‚17013274584906 bNDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.allocated‚341347506677760 XIDeviceStatsMonitor.on_validation_batch_start/active_bytes.large_pool.peak‚ 4541568000 PGDeviceStatsMonitor.on_validation_batch_start/allocation.large_pool.peak‚2460 RKDeviceStatsMonitor.on_validation_batch_start/inactive_split.large_pool.peak‚26 fRDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.large_pool.freed‚361673442465792 bTDeviceStatsMonitor.on_validation_batch_start/inactive_split_bytes.small_pool.current‚ 103378432 E=DeviceStatsMonitor.on_validation_batch_start/num_device_alloc‚416ßÅâ -2ƒ+- _wandbruntime‚9468  _runtime‚9468d*óÛ^)„+ØR TFDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.freed‚ 159678038 [LDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.current‚ 4624220160 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚17062807635456 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚379052326993408 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚324734153199616 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚341366353860608 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2460 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 244631888 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚341747758351498 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 145311232 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 404325905 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚361693046031360 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 410712064 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚985 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 140964952 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚17013622191242 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚17359384340480 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚324734136160256 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 5041553408 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚341366353860608 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1006 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚217 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 404309926 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 159678038 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚17062807635456 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2460 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚93214013 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚379052454311936 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚361693069971456 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚341366371863552 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚358429306807296 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 127308288 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚416 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 159678040 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 161830145 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 103378432 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 410712064 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚358429161496064 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚68616408 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚416 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 5041553408 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 127308288 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 161830421 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚17359280962048 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚416 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 404309926 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 4757245257 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚217 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 BDeviceStatsMonitor.on_validation_batch_end/segment.all.current‚416 [HDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.freed‚17062807635456 ]IDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.freed‚379052326993408 E?DeviceStatsMonitor.on_validation_batch_end/num_sync_all_streams‚2 KEDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.freed‚0 cODeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.allocated‚324734153199616 \HDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.freed‚341366353860608 PJDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.freed‚0 \NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.allocated‚ 417333248 JADeviceStatsMonitor.on_validation_batch_end/active.large_pool.peak‚2460 TFDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.freed‚ 244631888 OIDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.allocated‚0 XDDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.freed‚341747758351498 KADeviceStatsMonitor.on_validation_batch_end/active.small_pool.peak‚24128 QCDeviceStatsMonitor.on_validation_batch_end/active_bytes.all.current‚ 145311232 WIDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.small_pool.peak‚ 417333248 D:DeviceStatsMonitor.on_validation_batch_end/active.all.peak‚24401 MEDeviceStatsMonitor.on_validation_batch_end/segment.small_pool.current‚199 QCDeviceStatsMonitor.on_validation_batch_end/allocation.all.allocated‚ 404325905 dPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.freed‚361693046031360 YJDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.peak‚ 4390226944 XJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.peak‚ 410712064 KADeviceStatsMonitor.on_validation_batch_end/allocation.all.current‚15979 QIDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.peak‚985 TFDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.current‚ 140964952 ^KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.small_pool.freed‚17013622191242 WJDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.current‚18002944 gTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.allocated‚17359384340480 JDDeviceStatsMonitor.on_validation_batch_end/oversize_allocations.peak‚0 _KDeviceStatsMonitor.on_validation_batch_end/requested_bytes.large_pool.freed‚324734136160256 QBDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.peak‚ 5041553408 ]NDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.large_pool.allocated‚ 4624220160 _KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.freed‚341366353860608 KBDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.peak‚1006 GADeviceStatsMonitor.on_validation_batch_end/oversize_segments.peak‚0 ICDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.freed‚0 MEDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.current‚217 I;DeviceStatsMonitor.on_validation_batch_end/active.all.freed‚ 404309926 PBDeviceStatsMonitor.on_validation_batch_end/active.large_pool.freed‚ 159678038 ^KDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.freed‚17062807635456 H>DeviceStatsMonitor.on_validation_batch_end/allocation.all.peak‚24401 NEDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.peak‚2460 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.large_pool.allocated‚93214013 aMDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.all.allocated‚379052454311936 hTDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.allocated‚361693069971456 `LDeviceStatsMonitor.on_validation_batch_end/active_bytes.large_pool.allocated‚341366371863552 \HDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.allocated‚358429306807296 [MDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.small_pool.current‚ 127308288 C=DeviceStatsMonitor.on_validation_batch_end/num_oom_rejections‚0 C;DeviceStatsMonitor.on_validation_batch_end/num_device_alloc‚416 TFDeviceStatsMonitor.on_validation_batch_end/active.large_pool.allocated‚ 159678040 QCDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.freed‚ 161830145 `RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.current‚ 103378432 93DeviceStatsMonitor.on_validation_batch_end/num_ooms‚0 G=DeviceStatsMonitor.on_validation_batch_end/active.all.current‚15979 UGDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.peak‚ 410712064 XDDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.all.freed‚358429161496064 OEDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.peak‚24128 [NDeviceStatsMonitor.on_validation_batch_end/inactive_split.small_pool.allocated‚68616408 H@DeviceStatsMonitor.on_validation_batch_end/segment.all.allocated‚416 _RDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.large_pool.current‚23940096 TEDeviceStatsMonitor.on_validation_batch_end/reserved_bytes.all.current‚ 5041553408 XJDeviceStatsMonitor.on_validation_batch_end/active_bytes.small_pool.current‚ 127308288 UGDeviceStatsMonitor.on_validation_batch_end/inactive_split.all.allocated‚ 161830421 cPDeviceStatsMonitor.on_validation_batch_end/inactive_split_bytes.small_pool.freed‚17359280962048 C;DeviceStatsMonitor.on_validation_batch_end/segment.all.peak‚416 ICDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.freed‚0 YJDeviceStatsMonitor.on_validation_batch_end/allocated_bytes.large_pool.peak‚ 4541568000 M?DeviceStatsMonitor.on_validation_batch_end/allocation.all.freed‚ 404309926 RHDeviceStatsMonitor.on_validation_batch_end/allocation.small_pool.current‚15977 JDDeviceStatsMonitor.on_validation_batch_end/oversize_segments.current‚0 RCDeviceStatsMonitor.on_validation_batch_end/requested_bytes.all.peak‚ 4757245257 OGDeviceStatsMonitor.on_validation_batch_end/segment.large_pool.allocated‚217 NHDeviceStatsMonitor.on_validation_batch_end/allocation.large_pool.current‚2 B&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14072054624557495 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.022937189787626266 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1951247900724411 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.077175073325634 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20407360792160034 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.115079365670681 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.018386242911219597 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5517140626907349 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.007999495603144169 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_4_of_4‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11466648429632187 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13484284281730652 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18590931594371796 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_7_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9908352494239807 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11280249059200287 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3788760006427765 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1891242116689682 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9907435774803162 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35667985677719116 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1500408798456192 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9726383090019226 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.48431718349456787 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02083333395421505 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.41392773389816284 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 *jet/accuracy_1_of_3‚0.8658225536346436 ,particle/sensitivity‚0.30068957805633545 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8181818127632141 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 jet/accuracy_5_of_7‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.035535283386707306 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5278288125991821 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4174082577228546 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977049827575684 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0008818342466838658 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2688491940498352 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_2‚0.7553433775901794 particle/accuracy_3_of_4‚1 particle/accuracy_5_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9789473414421082 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.12888889014720917 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 jet/accuracy_6_of_7‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.41294804215431213 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4932386875152588 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.46000000834465027 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 +jet/accuracy_3_of_5‚0.08745519816875458 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5535714030265808 jet/accuracy_4_of_7‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_pur„íAuity‚0.6666666865348816 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.49541547894477844 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1453859657049179 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5458653569221497 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18333330750465393 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.0833333358168602 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14820219576358795 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5226680636405945 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 jet/accuracy_1_of_6‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2538914680480957 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.36079123616218567 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4746280014514923 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 +jet/accuracy_3_of_4‚0.07963959127664566 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5167264938354492 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5654616355895996 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.522988498210907 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.017786739394068718 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.22927990555763245 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 particle/accuracy_1_of_6‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.32499998807907104 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6666666865348816 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1562805473804474 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4918825924396515 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22371649742126465 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6064062714576721 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9845983386039734 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.0013020833721384406 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4943975508213043 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964737296104431 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.15909090638160706 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.47760844230651855 *jet/accuracy_2_of_4‚0.3798464834690094 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08674393594264984 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2042236477136612 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9767857193946838 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 *jet/accuracy_1_of_4‚0.9126855134963989 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5218731760978699 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1982758492231369 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03488372266292572 =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5873083472251892 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4924364984035492 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 (particle/f_score‚0.37788569927215576 jet/accuracy_1_of_7‚1 particle/accuracy_5_of_7‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.985336184501648 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3265080153942108 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0800037756562233 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5448632836341858 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1928648054599762 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 jet/accuracy_7_of_7‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_2_of_6‚0.5064102411270142 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5548437237739563 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5562776327133179 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9899340271949768 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_5‚1 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9880325794219971 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9927052855491638 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3134276270866394 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.43201756477355957 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267138838768005 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5080567598342896 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 +jet/accuracy_3_of_6‚0.14743588864803314 particle/accuracy_5_of_6‚1 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.2613636255264282 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_3‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ,"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.625 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2690586447715759 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4907498061656952 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9892290234565735 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26998311281204224 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29365089535713196 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5318278670310974 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_6‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1140558123588562 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 jet/accuracy_4_of_6‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5833647847175598 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26257967948913574 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5723521709442139 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04821428656578064 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22441928088665009 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23345328867435455 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19318801164627075 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17791444063186646 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 _step‚993 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9852967858314514 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.046226415783166885 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5836405754089355 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.48027724027633667 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0007936508045531809 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4109313488006592 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1593094915151596 particle/accuracy_3_of_3‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19326944649219513 6 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.135444775223732 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3727678656578064 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.09099617600440979 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4067564308643341 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998068630695343 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.35451388359069824 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5031111240386963 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.37105652689933777 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012415317818522453 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09219858795404434 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03191754221916199 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.18981480598449707 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 epoch‚9 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.40872931480407715 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4078544080257416 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.39684683084487915 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9935721158981323 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9864341616630554 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_2_of_7‚0 particle/accuracy_4_of_7‚1 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5374999642372131 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.34766313433647156 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5996527075767517 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35211125016212463 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3958333432674408 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5394283533096313 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8199805021286011 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08709890395402908 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5154321193695068 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3827160596847534 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15574471652507782 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_7‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.18582569062709808 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9917826056480408 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.024533886462450027 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.021820537745952606 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4136165678501129 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6952837705612183 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2777777910232544 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1589927226305008 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ,"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.625 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14918582141399384 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.558448076248169 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05921052768826485 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_6_of_6‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9847617745399475 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2083333283662796 particle/accuracy_4_of_5‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5603654980659485 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5316656231880188 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_5‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.18189655244350433 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0028083028737455606 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 jet/accuracy_3_of_7‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9942410588264465 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5538634657859802 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14421139657497406 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5664682984352112 +jet/accuracy_2_of_3‚0.24113234877586365 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17436589300632477 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 +jet/accuracy_2_of_5‚0.45112645626068115 particle/accuracy_1_of_4‚1 (particle/accuracy‚0.8455325961112976 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5917946696281433 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5091450214385986 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5127461552619934 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1652485877275467 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 particle/accuracy_6_of_7‚1 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_7‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 trainer/global_step‚12089 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17791444063186646 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 *loss/val_total_loss‚44.056250698250466 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9895954728126526 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1933611035346985 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.49515125155448914 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.48131969571113586 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.390316903591156 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5977563261985779 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.017237992957234383 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09923789650201797 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.31388890743255615 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14113369584083557 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.43947455286979675 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6519810557365417 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27617791295051575 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9786067008972168 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4079124331474304 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7878787517547607 particle/accuracy_2_of_2‚1 jet/accuracy_6_of_6‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5117378830909729 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5714560151100159 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.41590410470962524 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.44276103377342224 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009675396606326103 particle/accuracy_2_of_6‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5625 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.37092384696006775 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5212392807006836 *jet/accuracy_2_of_2‚0.1162157952785492 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5127461552619934 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.27129635214805603 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26519063115119934 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4056530296802521 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 $ _timestamp‚1.788404857020134e+09 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912897944450378 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9907218813896179 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.558625340461731 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21649058163166046 ,jet/accuracy_4_of_5‚0.004480286967009306 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.42227086424827576 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9662356376647949 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2847222089767456 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997404932975769 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1616325080394745 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9391193389892578 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.557440459728241 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03572239726781845 ,jet/accuracy_4_of_4‚0.006044226698577404 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.021859604865312576 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.23873873054981232 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 _runtime‚ 9481.65893146 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5117841958999634 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.469341516494751 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5462963581085205 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.19181284308433533 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0vÖf[± Š+Ϧ =&Purity/0FRt*SRqqt0FBt/event_proportion‚0.2192460298538208 9"Purity/0FRt*SRqqt2FBt/SRqqt_purity‚0.5873083472251892 7 Purity/1FRt*SRqqt2FBt/FRt_purity‚0.4924364984035492 ,&Purity/2FRt4SRqqt2FBt/event_proportion‚0 (particle/f_score‚0.37788569927215576 jet/accuracy_1_of_7‚1 particle/accuracy_5_of_7‚1 6 Purity/*FRt*SRqqt2FBt/FBt_purity‚0.985336184501648 ( Purity/*FRt*SRqqt3FBt/FRt_purity‚0.5 7 Purity/*FRt1SRqqt1FBt/FRt_purity‚0.3265080153942108 9"Purity/1FRt*SRqqt0FBt/event_purity‚0.0800037756562233 @&Purity/1FRt*SRqqt3FBt/event_proportion‚0.0013640873366966844 9"Purity/3FRt*SRqqt*FBt/SRqqt_purity‚0.5448632836341858 & Purity/1FRt3SRqqt1FBt/FBt_purity‚1 ("Purity/4FRt*SRqqt0FBt/event_purity‚0 7 Purity/2FRt1SRqqt*FBt/FRt_purity‚0.1928648054599762 7 Purity/3FRt2SRqqt1FBt/FRt_purity‚0.1428571492433548 jet/accuracy_7_of_7‚0 @&Purity/*FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt0FBt/event_proportion‚0.00012400794366840273 *jet/accuracy_2_of_6‚0.5064102411270142 9"Purity/*FRt2SRqqt1FBt/SRqqt_purity‚0.5548437237739563 7 Purity/1FRt0SRqqt2FBt/FRt_purity‚0.5562776327133179 7 Purity/1FRt1SRqqt*FBt/FBt_purity‚0.9899340271949768 @&Purity/1FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 @&Purity/3FRt*SRqqt2FBt/event_proportion‚0.0003100198518950492 particle/accuracy_1_of_5‚1 ,&Purity/4FRt2SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt4FBt/event_purity‚0.1666666716337204 :"Purity/*FRt0SRqqt4FBt/event_purity‚0.20000000298023224 & Purity/3FRt2SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt1SRqqt*FBt/FBt_purity‚0.9880325794219971 =&Purity/*FRt2SRqqt1FBt/event_proportion‚0.0225694440305233 9"Purity/3FRt2SRqqt1FBt/SRqqt_purity‚0.5714285969734192 ("Purity/3FRt3SRqqt*FBt/event_purity‚0 7 Purity/0FRt0SRqqt*FBt/FBt_purity‚0.9927052855491638 @&Purity/1FRt0SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/1FRt1SRqqt1FBt/FRt_purity‚0.3134276270866394 =&Purity/1FRt2SRqqt*FBt/event_proportion‚0.0573536716401577 7 Purity/1FRt2SRqqt1FBt/FBt_purity‚0.9793103337287903 8 Purity/2FRt0SRqqt2FBt/FRt_purity‚0.43201756477355957 ,&Purity/4FRt3SRqqt2FBt/event_proportion‚0 -"Purity/0FRt4SRqqt*FBt/SRqqt_purity‚0.3125 @&Purity/*FRt*SRqqt3FBt/event_proportion‚0.0045262896455824375 7 Purity/*FRt*SRqqt3FBt/FBt_purity‚0.9267138838768005 7 Purity/*FRt0SRqqt2FBt/FRt_purity‚0.5080567598342896 =&Purity/1FRt2SRqqt1FBt/event_proportion‚0.0088665671646595 ,&Purity/4FRt2SRqqt4FBt/event_proportion‚0 particle/accuracy_1_of_1‚1 +jet/accuracy_3_of_6‚0.14743588864803314 particle/accuracy_5_of_6‚1 7 Purity/2FRt1SRqqt2FBt/FRt_purity‚0.261363625526àëRù4282 @&Purity/2FRt4SRqqt*FBt/event_proportion‚0.0003100198518950492 particle/accuracy_2_of_3‚1 & Purity/3FRt1SRqqt*FBt/FBt_purity‚1 ,"Purity/3FRt3SRqqt0FBt/SRqqt_purity‚0.625 9"Purity/*FRt1SRqqt0FBt/event_purity‚0.2690586447715759 9"Purity/1FRt*SRqqt3FBt/event_purity‚0.1111111119389534 ,&Purity/4FRt*SRqqt1FBt/event_proportion‚0 *jet/accuracy_1_of_1‚0.4907498061656952 7 Purity/*FRt2SRqqt1FBt/FBt_purity‚0.9892290234565735 =&Purity/0FRt0SRqqt*FBt/event_proportion‚0.1191096231341362 8 Purity/1FRt0SRqqt*FBt/FRt_purity‚0.26998311281204224 :"Purity/1FRt0SRqqt1FBt/event_purity‚0.29365089535713196 9"Purity/2FRt2SRqqt*FBt/SRqqt_purity‚0.5318278670310974 ("Purity/*FRt4SRqqt*FBt/event_purity‚0 ("Purity/0FRt4SRqqt1FBt/event_purity‚0 particle/accuracy_4_of_6‚1 & Purity/*FRt3SRqqt1FBt/FBt_purity‚1 ,&Purity/0FRt3SRqqt3FBt/event_proportion‚0 A&Purity/1FRt4SRqqt0FBt/event_proportion‚0.00037202381645329297 ,&Purity/1FRt4SRqqt3FBt/event_proportion‚0 7 Purity/2FRt2SRqqt0FBt/FRt_purity‚0.1140558123588562 @&Purity/2FRt3SRqqt0FBt/event_proportion‚0.0035962301772087812 jet/accuracy_4_of_6‚0 & Purity/3FRt0SRqqt2FBt/FBt_purity‚1 9"Purity/*FRt1SRqqt1FBt/SRqqt_purity‚0.5833647847175598 =&Purity/0FRt2SRqqt0FBt/event_proportion‚0.0431547611951828 @&Purity/3FRt3SRqqt*FBt/event_proportion‚0.0005580357392318547 ,&Purity/3FRt4SRqqt4FBt/event_proportion‚0 7 Purity/3FRt1SRqqt1FBt/FRt_purity‚0.2499999850988388 & Purity/*FRt4SRqqt1FBt/FBt_purity‚1 ,"Purity/3FRt4SRqqt0FBt/SRqqt_purity‚0.125 =&Purity/*FRt1SRqqt0FBt/event_proportion‚0.3304191529750824 8 Purity/*FRt0SRqqt*FBt/FRt_purity‚0.26257967948913574 ,&Purity/*FRt4SRqqt3FBt/event_proportion‚0 9"Purity/0FRt1SRqqt2FBt/SRqqt_purity‚0.5723521709442139 :"Purity/2FRt0SRqqt1FBt/event_purity‚0.04821428656578064 ("Purity/2FRt0SRqqt3FBt/event_purity‚0 :"Purity/*FRt1SRqqt*FBt/event_purity‚0.22441928088665009 =&Purity/1FRt*SRqqt*FBt/event_proportion‚0.4355778694152832 8 Purity/2FRt0SRqqt*FBt/FRt_purity‚0.23345328867435455 ,&Purity/4FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/*FRt3SRqqt3FBt/event_proportion‚0 8 Purity/1FRt1SRqqt*FBt/FRt_purity‚0.19318801164627075 @&Purity/2FRt*SRqqt3FBt/event_proportion‚6.200397183420137e-05 & Purity/4FRt0SRqqt0FBt/FRt_purity‚0 @&Purity/0FRt2SRqqt2FBt/event_proportion‚0.0009920635493472219 :"Purity/1FRt0SRqqt0FBt/event_purity‚0.17791444063186646 ,&Purity/4FRt4SRqqt*FBt/event_proportion‚0 _step‚993 7 Purity/0FRt1SRqqt*FBt/FBt_purity‚0.9852967858314514 ;"Purity/0FRt2SRqqt1FBt/event_purity‚0.046226415783166885 ,"Purity/1FRt0SRqqt3FBt/event_purity‚0.125 <&Purity/1FRt1SRqqt*FBt/event_proportion‚0.172185018658638 7 Purity/2FRt1SRqqt*FBt/FBt_purity‚0.9937499761581421 ("Purity/0FRt3SRqqt0FBt/event_purity‚0 9"Purity/1FRt2SRqqt2FBt/SRqqt_purity‚0.4166666567325592 7 Purity/3FRt1SRqqt2FBt/FRt_purity‚0.3333333432674408 ("Purity/3FRt1SRqqt1FBt/event_purity‚0 7 Purity/3FRt4SRqqt*FBt/FRt_purity‚0.3333333432674408 + Purity/3FRt3SRqqt*FBt/FRt_purity‚0.1875 9"Purity/*FRt*SRqqt2FBt/SRqqt_purity‚0.5836405754089355 :"Purity/0FRt2SRqqt0FBt/SRqqt_purity‚0.48027724027633667 ?&Purity/0FRt3SRqqt0FBt/event_proportion‚0.006076388992369175 <"Purity/1FRt2SRqqt*FBt/event_purity‚0.0007936508045531809 ("Purity/3FRt2SRqqt0FBt/event_purity‚0 ("Purity/*FRt*SRqqt4FBt/SRqqt_purity‚1 9"Purity/*FRt3SRqqt*FBt/SRqqt_purity‚0.4109313488006592 9"Purity/1FRt*SRqqt1FBt/event_purity‚0.1593094915151596 particle/accuracy_3_of_3‚1 ("Purity/1FRt4SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt*FBt/FRt_purity‚0.19326944649219513 6 Purity/3FRt*SRqqt0FBt/FRt_purity‚0.135444775223732 ,&Purity/4FRt1SRqqt2FBt/event_proportion‚0 particle/accuracy_2_of_5‚1 *"Purity/3FRt1SRqqt2FBt/SRqqt_purity‚0.5 @&Purity/1FRt0SRqqt3FBt/event_proportion‚0.0011160714784637094 7 Purity/2FRt1SRqqt1FBt/FRt_purity‚0.3727678656578064 8 Purity/3FRt2SRqqt*FBt/FRt_purity‚0.09099617600440979 ("Purity/3FRt*SRqqt2FBt/event_purity‚0 7 Purity/*FRt0SRqqt1FBt/FRt_purity‚0.4067564308643341 6 Purity/1FRt*SRqqt1FBt/FBt_purity‚0.998068630695343 ,&Purity/3FRt2SRqqt4FBt/event_proportion‚0 =&Purity/0FRt*SRqqt1FBt/event_proportion‚0.1489335298538208 7 Purity/1FRt0SRqqt*FBt/FBt_purity‚0.9946674704551697 8 Purity/1FRt1SRqqt2FBt/FRt_purity‚0.35451388359069824 9"Purity/2FRt*SRqqt0FBt/SRqqt_purity‚0.5031111240386963 8 Purity/2FRt0SRqqt1FBt/FRt_purity‚0.37105652689933777 ("Purity/2FRt1SRqqt1FBt/event_purity‚0 :"Purity/*FRt2SRqqt2FBt/SRqqt_purity‚0.46666666865348816 ("Purity/1FRt1SRqqt3FBt/SRqqt_purity‚1 ;"Purity/1FRt1SRqqt0FBt/event_purity‚0.012415317818522453 & Purity/2FRt*SRqqt3FBt/FBt_purity‚1 9"Purity/0FRt1SRqqt3FBt/SRqqt_purity‚0.8571428656578064 :"Purity/*FRt*SRqqt3FBt/event_purity‚0.09219858795404434 :"Purity/*FRt2SRqqt*FBt/event_purity‚0.03191754221916199 ,&Purity/2FRt2SRqqt4FBt/event_proportion‚0 8 Purity/3FRt1SRqqt*FBt/FRt_purity‚0.18981480598449707 ("Purity/1FRt4SRqqt0FBt/event_purity‚0 9"Purity/*FRt1SRqqt3FBt/SRqqt_purity‚0.9444444179534912 epoch‚9 @&Purity/*FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 ,&Purity/0FRt4SRqqt4FBt/event_proportion‚0 A&Purity/1FRt*SRqqt4FBt/event_proportion‚0.00012400794366840273 8 Purity/1FRt0SRqqt1FBt/FRt_purity‚0.40872931480407715 9"Purity/1FRt3SRqqt*FBt/SRqqt_purity‚0.4078544080257416 7 Purity/2FRt0SRqqt*FBt/FBt_purity‚0.9976190328598022 :"Purity/2FRt3SRqqt*FBt/SRqqt_purity‚0.39684683084487915 ("Purity/3FRt2SRqqt*FBt/event_purity‚0 7 Purity/*FRt0SRqqt*FBt/FBt_purity‚0.9935721158981323 7 Purity/2FRt*SRqqt2FBt/FBt_purity‚0.9864341616630554 ,&Purity/4FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/4FRt4SRqqt2FBt/event_proportion‚0 ("Purity/*FRt4SRqqt1FBt/event_purity‚0 jet/accuracy_2_of_7‚0 particle/accuracy_4_of_7‚1 & Purity/2FRt*SRqqt3FBt/FRt_purity‚0 @&Purity/4FRt2SRqqt*FBt/event_proportion‚6.200397183420137e-05 9"Purity/1FRt2SRqqt1FBt/SRqqt_purity‚0.5374999642372131 ("Purity/1FRt0SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/SRqqt_purity‚1 ("Purity/4FRt0SRqqt*FBt/event_purity‚0 8 Purity/*FRt1SRqqt2FBt/FRt_purity‚0.34766313433647156 @&Purity/*FRt4SRqqt0FBt/event_proportion‚0.0012400794075801969 9"Purity/1FRt1SRqqt2FBt/SRqqt_purity‚0.5996527075767517 ?&Purity/1FRt3SRqqt*FBt/event_proportion‚0.008990575559437275 @&Purity/3FRt0SRqqt0FBt/event_proportion‚0.0016121031949296594 @&Purity/3FRt0SRqqt2FBt/event_proportion‚6.200397183420137e-05 * Purity/*FRt*SRqqt4FBt/FBt_purity‚0.875 & Purity/*FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/*FRt0SRqqt4FBt/event_proportion‚0.0003100198518950492 8 Purity/1FRt*SRqqt1FBt/FRt_purity‚0.35211125016212463 ("Purity/2FRt1SRqqt2FBt/event_purity‚0 <&Purity/0FRt*SRqqt2FBt/event_proportion‚0.039372518658638 ,&Purity/0FRt1SRqqt4FBt/event_proportion‚0 =&Purity/1FRt0SRqqt1FBt/event_proportion‚0.0495411716401577 & Purity/2FRt0SRqqt1FBt/FBt_purity‚1 @&Purity/2FRt2SRqqt2FBt/event_proportion‚6.200397183420137e-05 @&Purity/3FRt1SRqqt*FBt/event_proportion‚0.0037202381063252687 9"Purity/1FRt3SRqqt1FBt/SRqqt_purity‚0.3958333432674408 7 Purity/1FRt2SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt1SRqqt*FBt/SRqqt_purity‚0.5394283533096313 9"Purity/0FRt0SRqqt*FBt/event_purity‚0.8199805021286011 4&Purity/1FRt3SRqqt0FBt/event_proportion‚ 0.0078125 ,&Purity/2FRt4SRqqt1FBt/event_proportion‚0 & Purity/3FRt*SRqqt2FBt/FBt_purity‚1 :"Purity/0FRt2SRqqt0FBt/event_purity‚0.08709890395402908 =&Purity/1FRt*SRqqt2FBt/event_proportion‚0.0223214291036129 9"Purity/3FRt2SRqqt0FBt/SRqqt_purity‚0.5154321193695068 9"Purity/*FRt3SRqqt1FBt/SRqqt_purity‚0.3827160596847534 7 Purity/*FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 ) Purity/4FRt1SRqqt0FBt/FRt_purity‚0.25 8 Purity/*FRt2SRqqt*FBt/FRt_purity‚0.15574471652507782 & Purity/1FRt0SRqqt1FBt/FBt_purity‚1 >&Purity/2FRt*SRqqt1FBt/event_proportion‚0.02114335261285305 @&Purity/3FRt0SRqqt*FBt/event_proportion‚0.0028521826025098562 @&Purity/4FRt0SRqqt0FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_1_of_7‚1 @&Purity/*FRt1SRqqt3FBt/event_proportion‚0.0006820436683483422 8 Purity/*FRt3SRqqt*FBt/FRt_purity‚0.18582569062709808 @&Purity/0FRt3SRqqt*FBt/event_proportion‚0.0066964286379516125 7 Purity/1FRt*SRqqt*FBt/FBt_purity‚0.9917826056480408 ;"Purity/2FRt*SRqqt1FBt/event_purity‚0.024533886462450027 ("Purity/2FRt2SRqqt1FBt/event_purity‚0 ,jet/accuracy_3_of_3‚0.021820537745952606 9"Purity/0FRt3SRqqt0FBt/SRqqt_purity‚0.4136165678501129 9"Purity/*FRt0SRqqt1FBt/event_purity‚0.6952837705612183 =&Purity/0FRt1SRqqt*FBt/event_proportion‚0.2310267835855484 ,&Purity/1FRt4SRqqt1FBt/event_proportion‚0 ,&Purity/2FRt2SRqqt3FBt/event_proportion‚0 ?&Purity/3FRt0SRqqt1FBt/event_proportion‚0.001178075443021953 9"Purity/0FRt3SRqqt1FBt/SRqqt_purity‚0.3888888955116272 ("Purity/2FRt2SRqqt*FBt/event_purity‚0 9"Purity/*FRt4SRqqt*FBt/SRqqt_purity‚0.2777777910232544 7 Purity/1FRt4SRqqt*FBt/FRt_purity‚0.1666666716337204 & Purity/1FRt3SRqqt2FBt/FRt_purity‚0 ("Purity/0FRt0SRqqt1FBt/event_purity‚1 7 Purity/1FRt2SRqqt*FBt/FRt_purity‚0.1589927226305008 @&Purity/1FRt3SRqqt2FBt/event_proportion‚6.200397183420137e-05 particle/accuracy_3_of_6‚1 ("Purity/0FRt2SRqqt2FBt/event_purity‚0 ,"Purity/3FRt3SRqqt*FBt/SRqqt_purity‚0.625 ("Purity/*FRt3SRqqt2FBt/event_purity‚0 9"Purity/1FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 8 Purity/2FRt*SRqqt0FBt/FRt_purity‚0.14918582141399384 ?&Purity/2FRt2SRqqt0FBt/event_proportion‚0.015376984141767025 @&Purity/3FRt*SRqqt1FBt/event_proportion‚0.0026041667442768812 ,&Purity/3FRt4SRqqt2FBt/event_proportion‚0 ("Purity/1FRt1SRqqt4FBt/event_purity‚0 *"Purity/*FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/2FRt2SRqqt2FBt/SRqqt_purity‚0 =&Purity/*FRt0SRqqt1FBt/event_proportion‚0.1462053507566452 8"Purity/0FRt*SRqqt1FBt/SRqqt_purity‚0.558448076248169 :"Purity/2FRt0SRqqt2FBt/event_purity‚0.05921052768826485 ,&Purity/2FRt1SRqqt3FBt/event_proportion‚0 @&Purity/3FRt3SRqqt0FBt/event_proportion‚0.0005580357392318547 7 Purity/0FRt*SRqqt3FBt/FBt_purity‚0.9122807383537292 ("Purity/0FRt4SRqqt*FBt/event_purity‚0 particle/accuracy_6_of_6‚1 7 Purity/0FRt*SRqqt2FBt/FBt_purity‚0.9847617745399475 7 Purity/2FRt3SRqqt0FBt/FRt_purity‚0.2083333283662796 particle/accuracy_4_of_5‚1 9"Purity/1FRt4SRqqt0FBt/SRqqt_purity‚0.2916666567325592 & Purity/*FRt3SRqqt2FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt1FBt/SRqqt_purity‚0.5603654980659485 9"Purity/1FRt1SRqqt0FBt/SRqqt_purity‚0.5316656231880188 ("Purity/3FRt*SRqqt*FBt/event_purity‚0 jet/accuracy_5_of_5‚0 :"Purity/0FRt*SRqqt3FBt/event_purity‚0.07894736528396606 8 Purity/2FRt4SRqqt*FBt/FRt_purity‚0.20000000298023224 , Purity/*FRt2SRqqt2FBt/FRt_purity‚0.34375 ,&Purity/*FRt4SRqqt2FBt/event_proportion‚0 8 Purity/1FRt3SRqqt*FBt/FRt_purity‚0.18189655244350433 <"Purity/2FRt*SRqqt0FBt/event_purity‚0.0028083028737455606 >&Purity/2FRt0SRqqt0FBt/event_proportion‚0.02480158768594265 jet/accuracy_3_of_7‚0 7 Purity/0FRt1SRqqt1FBt/FBt_purity‚0.9942410588264465 & Purity/2FRt1SRqqt1FBt/FBt_purity‚1 particle/accuracy_2_of_7‚1 9"Purity/2FRt*SRqqt1FBt/SRqqt_purity‚0.5538634657859802 8 Purity/2FRt2SRqqt*FBt/FRt_purity‚0.14421139657497406 9"Purity/2FRt2SRqqt1FBt/SRqqt_purity‚0.5664682984352112 +jet/accuracy_2_of_3‚0.24113234877586365 ("Purity/2FRt4SRqqt0FBt/event_purity‚0 ("Purity/4FRt2SRqqt0FBt/event_purity‚0 ,&Purity/0FRt3SRqqt4FBt/event_proportion‚0 >&Purity/2FRt*SRqqt2FBt/event_proportion‚0.00533234141767025 @&Purity/2FRt0SRqqt3FBt/event_proportion‚6.200397183420137e-05 =&Purity/2FRt2SRqqt*FBt/event_proportion‚0.0193452388048172 @&Purity/3FRt1SRqqt0FBt/event_proportion‚0.0024801588151603937 ,&Purity/3FRt1SRqqt4FBt/event_proportion‚0 8 Purity/*FRt0SRqqt0FBt/FRt_purity‚0.17436589300632477 ,&Purity/*FRt3SRqqt4FBt/event_proportion‚0 A&Purity/0FRt0SRqqt4FBt/event_proportion‚0.00024801588733680546 ("Purity/1FRt3SRqqt0FBt/event_purity‚0 @&Purity/3FRt1SRqqt1FBt/event_proportion‚0.0009920635493472219 +jet/accuracy_2_of_5‚0.45112645626068115 particle/accuracy_1_of_4‚1 (particle/accuracy‚0.8455325961112976 9"Purity/*FRt1SRqqt2FBt/SRqqt_purity‚0.5917946696281433 9"Purity/0FRt*SRqqt*FBt/SRqqt_purity‚0.5091450214385986 =&Purity/0FRt0SRqqt0FBt/event_proportion‚0.0407366082072258 9"Purity/0FRt1SRqqt0FBt/SRqqt_purity‚0.5127461552619934 7 Purity/2FRt0SRqqt0FBt/FRt_purity‚0.1652485877275467 @&Purity/3FRt2SRqqt0FBt/event_proportion‚0.0024801588151603937 +"Purity/0FRt*SRqqt4FBt/event_purity‚0.25 particle/accuracy_6_of_7‚1 =&Purity/1FRt2SRqqt0FBt/event_proportion‚0.0474950410425663 =&Purity/2FRt*SRqqt0FBt/event_proportion‚0.0758928582072258 ("Purity/2FRt1SRqqt0FBt/event_purity‚0 ,&Purity/3FRt2SRqqt2FBt/event_proportion‚0 ("Purity/1FRt3SRqqt1FBt/event_purity‚0 particle/accuracy_3_of_7‚1 ("Purity/0FRt3SRqqt1FBt/event_purity‚0 trainer/global_step‚12089 ,&Purity/0FRt4SRqqt3FBt/event_proportion‚0 @&Purity/4FRt1SRqqt*FBt/event_proportion‚6.200397183420137e-05 ("Purity/4FRt2SRqqt*FBt/SRqqt_purity‚1 8 Purity/1FRt0SRqqt0FBt/FRt_purity‚0.17791444063186646 >&Purity/2FRt0SRqqt2FBt/event_proportion‚0.00341021828353405 ("Purity/*FRt3SRqqt1FBt/event_purity‚0 *loss/val_total_loss‚44.056250698250466 7 Purity/0FRt*SRqqt*FBt/FBt_purity‚0.9895954728126526 ,&Purity/3FRt3SRqqt2FBt/event_proportion‚0 9"Purity/0FRt1SRqqt1FBt/event_purity‚0.1933611035346985 :"Purity/0FRt2SRqqt*FBt/SRqqt_purity‚0.49515125155448914 :"Purity/*FRt2SRqqt0FBt/SRqqt_purity‚0.48131969571113586 <&Purity/0FRt1SRqqt1FBt/event_proportion‚0.052269347012043 =&Purity/1FRt*SRqqt0FBt/event_proportion‚0.3129960298538208 7 Purity/1FRt*SRqqt3FBt/FBt_purity‚0.9444444179534912 & Purity/3FRt*SRqqt*FBt/FBt_purity‚1 & Purity/0FRt4SRqqt*FBt/FBt_purity‚1 ( Purity/2FRt2SRqqt2FBt/FRt_purity‚0.5 8"Purity/*FRt0SRqqt*FBt/event_purity‚0.390316903591156 =&Purity/*FRt0SRqqt2FBt/event_proportion‚0.0482390858232975 ,&Purity/1FRt2SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt0SRqqt1FBt/event_proportion‚0 =&Purity/0FRt1SRqqt2FBt/event_proportion‚0.0087425597012043 9"Purity/1FRt*SRqqt2FBt/SRqqt_purity‚0.5977563261985779 ;"Purity/1FRt1SRqqt*FBt/event_purity‚0.017237992957234383 7 Purity/2FRt*SRqqt1FBt/FBt_purity‚0.9960317611694336 ,&Purity/4FRt4SRqqt3FBt/event_proportion‚0 & Purity/*FRt0SRqqt4FBt/FRt_purity‚0 -"Purity/0FRt*SRqqt3FBt/SRqqt_purity‚0.8125 :"Purity/1FRt*SRqqt*FBt/event_purity‚0.09923789650201797 ,&Purity/3FRt3SRqqt3FBt/event_proportion‚0 8 Purity/3FRt*SRqqt1FBt/FRt_purity‚0.31388890743255615 ("Purity/3FRt1SRqqt*FBt/event_purity‚0 & Purity/0FRt3SRqqt1FBt/FBt_purity‚1 >&Purity/0FRt0SRqqt2FBt/event_proportion‚0.02963789738714695 =&Purity/2FRt1SRqqt0FBt/event_proportion‚0.0318080373108387 8 Purity/2FRt1SRqqt0FBt/FRt_purity‚0.14113369584083557 @&Purity/2FRt1SRqqt2FBt/event_proportion‚0.0018601190531626344 ,&Purity/3FRt*SRqqt4FBt/event_proportion‚0 7validation_average_jet_accuracy‚0.43947455286979675 9"Purity/0FRt*SRqqt1FBt/event_purity‚0.6519810557365417 :"Purity/0FRt*SRqqt2FBt/event_purity‚0.27617791295051575 7 Purity/1FRt*SRqqt2FBt/FBt_purity‚0.9786067008972168 7 Purity/1FRt*SRqqt3FBt/FRt_purity‚0.5277777910232544 @&Purity/1FRt3SRqqt1FBt/event_proportion‚0.0011160714784637094 =&Purity/2FRt*SRqqt*FBt/event_proportion‚0.1024305522441864 9"Purity/2FRt3SRqqt0FBt/SRqqt_purity‚0.4079124331474304 ("Purity/*FRt4SRqqt0FBt/event_purity‚0 >&Purity/1FRt0SRqqt2FBt/event_proportion‚0.01512896828353405 9"Purity/2FRt*SRqqt2FBt/SRqqt_purity‚0.7878787517547607 particle/accuracy_2_of_2‚1 jet/accuracy_6_of_6‚0 9"Purity/0FRt4SRqqt0FBt/SRqqt_purity‚0.3214285671710968 & Purity/2FRt3SRqqt1FBt/FBt_purity‚1 & Purity/1FRt0SRqqt4FBt/FRt_purity‚0 ("Purity/*FRt2SRqqt3FBt/event_purity‚0 9"Purity/*FRt*SRqqt*FBt/SRqqt_purity‚0.5117378830909729 9"Purity/1FRt*SRqqt1FBt/SRqqt_purity‚0.5714560151100159 :"Purity/0FRt3SRqqt*FBt/SRqqt_purity‚0.41590410470962524 & Purity/2FRt0SRqqt3FBt/FBt_purity‚1 * Purity/4FRt*SRqqt0FBt/FRt_purity‚0.125 8 Purity/*FRt*SRqqt2FBt/FRt_purity‚0.44276103377342224 7 Purity/*FRt0SRqqt3FBt/FBt_purity‚0.9404761791229248 ;"Purity/2FRt*SRqqt*FBt/event_purity‚0.009675396606326103 particle/accuracy_2_of_6‚1 & Purity/0FRt3SRqqt*FBt/FBt_purity‚1 ) Purity/*FRt1SRqqt4FBt/FBt_purity‚0.75 =&Purity/*FRt*SRqqt1FBt/event_proportion‚0.2714533805847168 @&Purity/*FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 @&Purity/0FRt3SRqqt1FBt/event_proportion‚0.0006200397037900984 7 Purity/1FRt0SRqqt2FBt/FBt_purity‚0.9923497438430786 + Purity/1FRt0SRqqt3FBt/FRt_purity‚0.5625 ,&Purity/2FRt4SRqqt3FBt/event_proportion‚0 particle/accuracy_1_of_2‚1 * Purity/0FRt0SRqqt4FBt/FBt_purity‚0.875 8 Purity/2FRt*SRqqt1FBt/FRt_purity‚0.37092384696006775 9"Purity/2FRt1SRqqt0FBt/SRqqt_purity‚0.5212392807006836 *jet/accuracy_2_of_2‚0.1162157952785492 & Purity/2FRt3SRqqt*FBt/FBt_purity‚1 & Purity/1FRt*SRqqt4FBt/FRt_purity‚0 ("Purity/4FRt*SRqqt*FBt/event_purity‚0 ?&Purity/*FRt3SRqqt1FBt/event_proportion‚0.002108135027810931 9"Purity/0FRt1SRqqt0FBt/event_purity‚0.5127461552619934 =&Purity/1FRt*SRqqt1FBt/event_proportion‚0.0987723246216774 & Purity/3FRt*SRqqt1FBt/FBt_purity‚1 8 Purity/3FRt0SRqqt*FBt/FRt_purity‚0.27129635214805603 & Purity/0FRt4SRqqt1FBt/FBt_purity‚1 :"Purity/*FRt0SRqqt2FBt/event_purity‚0.26519063115119934 7 Purity/1FRt0SRqqt3FBt/FBt_purity‚0.9583333134651184 9"Purity/1FRt3SRqqt0FBt/SRqqt_purity‚0.4056530296802521 @&Purity/4FRt0SRqqt*FBt/event_proportion‚6.200397183420137e-05 ,&Purity/4FRt3SRqqt0FBt/event_proportion‚0 particle/accuracy_1_of_3‚1 ("Purity/*FRt1SRqqt4FBt/SRqqt_purity‚1 $ _timestamp‚1.788404857020134e+09 7 Purity/*FRt0SRqqt2FBt/FBt_purity‚0.9912897944450378 6 Purity/*FRt0SRqqt3FBt/FRt_purity‚0.529411792755127 7 Purity/2FRt*SRqqt*FBt/FBt_purity‚0.9907218813896179 ,&Purity/4FRt*SRqqt4FBt/event_proportion‚0 -"Purity/3FRt1SRqqt1FBt/SRqqt_purity‚0.6875 7 Purity/2FRt3SRqqt1FBt/FRt_purity‚0.4166666567325592 8"Purity/0FRt2SRqqt1FBt/SRqqt_purity‚0.558625340461731 7 Purity/2FRt0SRqqt2FBt/FBt_purity‚0.9956141114234924 ("Purity/2FRt1SRqqt*FBt/event_purity‚0 :"Purity/2FRt4SRqqt0FBt/SRqqt_purity‚0.30000001192092896 & Purity/2FRt0SRqqt3FBt/FRt_purity‚0 ) Purity/1FRt1SRqqt4FBt/FBt_purity‚0.75 ("Purity/4FRt2SRqqt0FBt/SRqqt_purity‚1 8 Purity/*FRt*SRqqt*FBt/FRt_purity‚0.21649058163166046 ,jet/accuracy_4_of_5‚0.004480286967009306 +"Purity/*FRt4SRqqt1FBt/SRqqt_purity‚0.25 ("Purity/3FRt4SRqqt*FBt/event_purity‚0 :"Purity/0FRt1SRqqt*FBt/event_purity‚0.42227086424827576 7 Purity/0FRt1SRqqt2FBt/FBt_purity‚0.9662356376647949 >&Purity/0FRt2SRqqt1FBt/event_proportion‚0.00936259888112545 @&Purity/1FRt1SRqqt4FBt/event_proportion‚6.200397183420137e-05 7 Purity/2FRt1SRqqt2FBt/FBt_purity‚0.9659090638160706 ("Purity/2FRt3SRqqt0FBt/event_purity‚0 9"Purity/*FRt4SRqqt0FBt/SRqqt_purity‚0.2847222089767456 ("Purity/0FRt4SRqqt0FBt/event_purity‚0 6 Purity/0FRt*SRqqt1FBt/FBt_purity‚0.997404932975769 7 Purity/1FRt*SRqqt0FBt/FRt_purity‚0.1616325080394745 ,&Purity/1FRt3SRqqt3FBt/event_proportion‚0 *jet/accuracy_1_of_5‚0.9391193389892578 =&Purity/1FRt1SRqqt0FBt/event_proportion‚0.1264880895614624 ("Purity/1FRt2SRqqt1FBt/event_purity‚0 ("Purity/2FRt4SRqqt*FBt/event_purity‚0 8"Purity/2FRt1SRqqt1FBt/SRqqt_purity‚0.557440459728241 ?&Purity/3FRt*SRqqt0FBt/event_proportion‚0.007254464086145163 ,&Purity/4FRt0SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt1FBt/event_purity‚0 ("Purity/0FRt1SRqqt3FBt/event_purity‚0 ,&Purity/4FRt3SRqqt3FBt/event_proportion‚0 7 Purity/3FRt0SRqqt2FBt/FRt_purity‚0.3333333432674408 9"Purity/*FRt*SRqqt3FBt/SRqqt_purity‚0.8999999761581421 ("Purity/*FRt1SRqqt3FBt/event_purity‚0 ("Purity/1FRt*SRqqt4FBt/SRqqt_purity‚1 & Purity/4FRt2SRqqt*FBt/FRt_purity‚0 :"Purity/*FRt2SRqqt0FBt/event_purity‚0.03572239726781845 ,jet/accuracy_4_of_4‚0.006044226698577404 7 Purity/0FRt1SRqqt3FBt/FBt_purity‚0.8571428656578064 ;"Purity/0FRt1SRqqt2FBt/event_purity‚0.021859604865312576 =&Purity/2FRt0SRqqt1FBt/event_proportion‚0.0088665671646595 ("Purity/1FRt1SRqqt3FBt/event_purity‚0 @&Purity/0FRt4SRqqt0FBt/event_proportion‚0.0004340277810115367 8 Purity/2FRt3SRqqt*FBt/FRt_purity‚0.23873873054981232 ("Purity/3FRt2SRqqt1FBt/event_purity‚0 _runtime‚ 9481.65893146 <&Purity/*FRt*SRqqt2FBt/event_proportion‚0.067336305975914 9"Purity/1FRt*SRqqt*FBt/SRqqt_purity‚0.5117841958999634 8"Purity/1FRt2SRqqt0FBt/SRqqt_purity‚0.469341516494751 9"Purity/3FRt1SRqqt*FBt/SRqqt_purity‚0.5462963581085205 8 Purity/2FRt4SRqqt0FBt/FRt_purity‚0.20000000298023224 8 Purity/1FRt3SRqqt0FBt/FRt_purity‚0.19181284308433533 ,&Purity/4FRt4SRqqt0FBt/event_proportion‚0 7 Purity/3FRt0SRqqt1FBt/FRt_purity‚0.4444445073604584 jet/accuracy_5_of_6‚0 ("Purity/0FRt0SRqqt0FBt/event_purity‚1 +particle/specificity‚0.9463761448860168 ("Purity/1FRt2SRqqt2FBt/event_purity‚0 & Purity/3FRt1SRqqt2FBt/FBt_purity‚1 =&Purity/*FRt*SRqqt*FBt/event_proportion‚0.9592633843421936 >&Purity/*FRt3SRqqt0FBt/event_proportion‚0.01804315485060215 =&Purity/0FRt*SRqqt*FBt/event_proportion‚0.4109002947807312 8 Purity/1FRt2SRqqt0FBt/FRt_purity‚0.14072054624557495 ;"Purity/2FRt0SRqqt*FBt/event_purity‚0.022937189787626266 9"Purity/*FRt*SRqqt2FBt/event_purity‚0.1951247900724411 8"Purity/0FRt2SRqqt*FBt/event_purity‚0.077175073325634 8 Purity/2FRt*SRqqt*FBt/FRt_purity‚0.20407360792160034 ("Purity/*FRt1SRqqt4FBt/event_purity‚0 ("Purity/4FRt1SRqqt0FBt/event_purity‚0 8"Purity/*FRt0SRqqt3FBt/event_purity‚0.115079365670681 ;"Purity/*FRt2SRqqt1FBt/event_purity‚0.018386242911219597 ,&Purity/0FRt4SRqqt2FBt/event_proportion‚0 9"Purity/1FRt1SRqqt*FBt/SRqqt_purity‚0.5517140626907349 ,&Purity/2FRt3SRqqt4FBt/event_proportion‚0 A&Purity/4FRt*SRqqt0FBt/event_proportion‚0.00018601190822664648 *"Purity/3FRt*SRqqt2FBt/SRqqt_purity‚0.5 ;"Purity/2FRt0SRqqt0FBt/event_purity‚0.007999495603144169 >&Purity/3FRt*SRqqt*FBt/event_proportion‚0.01016865111887455 particle/accuracy_4_of_4‚1 & Purity/3FRt0SRqqt*FBt/FBt_purity‚1 ("Purity/3FRt1SRqqt2FBt/event_purity‚0 :"Purity/*FRt1SRqqt1FBt/event_purity‚0.11466648429632187 8 Purity/*FRt2SRqqt0FBt/FRt_purity‚0.13484284281730652 ,&Purity/0FRt2SRqqt4FBt/event_proportion‚0 @&Purity/0FRt4SRqqt*FBt/event_proportion‚0.0004960317746736109 =&Purity/1FRt1SRqqt1FBt/event_proportion‚0.0392485111951828 7 Purity/1FRt1SRqqt2FBt/FBt_purity‚0.9739583134651184 ,&Purity/1FRt3SRqqt4FBt/event_proportion‚0 ("Purity/2FRt2SRqqt0FBt/event_purity‚0 8 Purity/*FRt3SRqqt0FBt/FRt_purity‚0.18590931594371796 @&Purity/0FRt*SRqqt3FBt/event_proportion‚0.0031001984607428312 7 Purity/*FRt0SRqqt4FBt/FBt_purity‚0.8999999761581421 ("Purity/*FRt2SRqqt2FBt/event_purity‚0 particle/accuracy_7_of_7‚1 7 Purity/*FRt1SRqqt3FBt/FBt_purity‚0.8888888955116272 ) Purity/1FRt1SRqqt3FBt/FRt_purity‚0.25 @&Purity/*FRt2SRqqt2FBt/event_proportion‚0.0020461310632526875 5 Purity/0FRt2SRqqt1FBt/FBt_purity‚0.99622642993927 ("Purity/3FRt*SRqqt0FBt/event_purity‚0 ,&Purity/3FRt4SRqqt1FBt/event_proportion‚0 7 Purity/0FRt0SRqqt3FBt/FBt_purity‚0.9375000596046448 7 Purity/*FRt*SRqqt*FBt/FBt_purity‚0.9908352494239807 :"Purity/1FRt0SRqqt2FBt/event_purity‚0.11280249059200287 =&Purity/2FRt1SRqqt*FBt/event_proportion‚0.0416666679084301 ,&Purity/3FRt*SRqqt3FBt/event_proportion‚0 7 Purity/1FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 7 Purity/2FRt*SRqqt2FBt/FRt_purity‚0.3788760006427765 ?&Purity/2FRt3SRqqt*FBt/event_proportion‚0.003968254197388887 A&Purity/2FRt3SRqqt1FBt/event_proportion‚0.00037202381645329297 ,&Purity/3FRt0SRqqt3FBt/event_proportion‚0 ,&Purity/3FRt2SRqqt3FBt/event_proportion‚0 9"Purity/*FRt*SRqqt0FBt/event_purity‚0.1891242116689682 7 Purity/0FRt0SRqqt2FBt/FBt_purity‚0.9907435774803162 ("Purity/1FRt*SRqqt4FBt/event_purity‚0 & Purity/1FRt0SRqqt4FBt/FBt_purity‚1 @&Purity/4FRt1SRqqt0FBt/event_proportion‚6.200397183420137e-05 8 Purity/*FRt*SRqqt1FBt/FRt_purity‚0.35667985677719116 9"Purity/*FRt0SRqqt0FBt/event_purity‚0.1500408798456192 7 Purity/*FRt1SRqqt2FBt/FBt_purity‚0.9726383090019226 :"Purity/0FRt*SRqqt*FBt/event_purity‚0.48431718349456787 =&Purity/1FRt0SRqqt*FBt/event_proportion‚0.1966765820980072 :"Purity/1FRt1SRqqt2FBt/event_purity‚0.02083333395421505 ,&Purity/1FRt4SRqqt2FBt/event_proportion‚0 :"Purity/0FRt*SRqqt0FBt/event_purity‚0.41392773389816284 @&Purity/*FRt0SRqqt3FBt/event_proportion‚0.0037822420708835125 ("Purity/*FRt3SRqqt0FBt/event_purity‚0 @&Purity/0FRt0SRqqt3FBt/event_proportion‚0.0026041667442768812 =&Purity/0FRt1SRqqt0FBt/event_proportion‚0.1695808470249176 ,&Purity/2FRt0SRqqt4FBt/event_proportion‚0 6 Purity/2FRt2SRqqt1FBt/FBt_purity‚0.976190447807312 ,&Purity/4FRt1SRqqt4FBt/event_proportion‚0 =&Purity/*FRt2SRqqt*FBt/event_proportion‚0.1332465261220932 *jet/accuracy_1_of_3‚0.8658225536346436 ,particle/sensitivity‚0.30068957805633545 /"Purity/0FRt0SRqqt3FBt/event_purity‚0.109375 ("Purity/3FRt1SRqqt0FBt/event_purity‚0 7 Purity/*FRt4SRqqt0FBt/FRt_purity‚0.1666666716337204 +"Purity/0FRt4SRqqt1FBt/SRqqt_purity‚0.25 7 Purity/1FRt2SRqqt2FBt/FBt_purity‚0.9666666388511658 =&Purity/*FRt0SRqqt0FBt/event_proportion‚0.1573040634393692 9"Purity/2FRt1SRqqt2FBt/SRqqt_purity‚0.8181818127632141 ,&Purity/2FRt3SRqqt3FBt/event_proportion‚0 particle/accuracy_2_of_4‚1 * Purity/1FRt*SRqqt4FBt/FBt_purity‚0.875 ) Purity/*FRt1SRqqt3FBt/FRt_purity‚0.25 =&Purity/3FRt2SRqqt*FBt/event_proportion‚0.0029141865670681 jet/accuracy_5_of_7‚0 ("Purity/1FRt*SRqqt3FBt/SRqqt_purity‚1 7 Purity/1FRt1SRqqt3FBt/FBt_purity‚0.8333333730697632 ) Purity/4FRt1SRqqt*FBt/FRt_purity‚0.25 ("Purity/4FRt1SRqqt0FBt/SRqqt_purity‚1 ("Purity/4FRt2SRqqt*FBt/event_purity‚0 ;"Purity/1FRt1SRqqt1FBt/event_purity‚0.035535283386707306 7 Purity/2FRt2SRqqt*FBt/FBt_purity‚0.9767441749572754 A&Purity/3FRt1SRqqt2FBt/event_proportion‚0.00024801588733680546 ,&Purity/3FRt4SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt3FBt/event_proportion‚0 7 Purity/3FRt*SRqqt2FBt/FRt_purity‚0.3333333432674408 & Purity/*FRt1SRqqt4FBt/FRt_purity‚0 9"Purity/0FRt1SRqqt*FBt/SRqqt_purity‚0.5278288125991821 @&Purity/0FRt2SRqqt3FBt/event_proportion‚6.200397183420137e-05 ,&Purity/1FRt4SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt1FBt/event_purity‚0.4174082577228546 @&Purity/*FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 =&Purity/0FRt0SRqqt1FBt/event_proportion‚0.0866195410490036 ,&Purity/2FRt3SRqqt2FBt/event_proportion‚0 ,&Purity/3FRt1SRqqt3FBt/event_proportion‚0 ,&Purity/4FRt*SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt2SRqqt3FBt/event_proportion‚0 9"Purity/2FRt3SRqqt1FBt/SRqqt_purity‚0.3888888657093048 =&Purity/*FRt1SRqqt*FBt/event_proportion‚0.4486607015132904 7 Purity/*FRt*SRqqt1FBt/FBt_purity‚0.9977049827575684 <"Purity/1FRt2SRqqt0FBt/event_purity‚0.0008818342466838658 7 Purity/2FRt2SRqqt1FBt/FRt_purity‚0.2688491940498352 @&Purity/4FRt2SRqqt0FBt/event_proportion‚6.200397183420137e-05 *jet/accuracy_1_of_2‚0.7553433775901794 particle/accuracy_3_of_4‚1 particle/accuracy_5_of_5‚1 =&Purity/*FRt0SRqqt*FBt/event_proportion‚0.3558407723903656 7 Purity/0FRt2SRqqt*FBt/FBt_purity‚0.9789473414421082 ,&Purity/2FRt*SRqqt4FBt/event_proportion‚0 ("Purity/3FRt*SRqqt1FBt/event_purity‚0 8 Purity/3FRt1SRqqt0FBt/FRt_purity‚0.12888889014720917 :"Purity/2FRt4SRqqt*FBt/SRqqt_purity‚0.30000001192092896 jet/accuracy_6_of_7‚0 & Purity/1FRt3SRqqt2FBt/FBt_purity‚1 :"Purity/*FRt3SRqqt0FBt/SRqqt_purity‚0.41294804215431213 ,&Purity/*FRt4SRqqt4FBt/event_proportion‚0 9"Purity/0FRt*SRqqt0FBt/SRqqt_purity‚0.4932386875152588 @&Purity/0FRt4SRqqt1FBt/event_proportion‚6.200397183420137e-05 A&Purity/3FRt4SRqqt*FBt/event_proportion‚0.00012400794366840273 ,&Purity/4FRt2SRqqt2FBt/event_proportion‚0 :"Purity/3FRt1SRqqt0FBt/SRqqt_purity‚0.46000000834465027 ,"Purity/3FRt4SRqqt*FBt/SRqqt_purity‚0.125 =&Purity/*FRt1SRqqt1FBt/event_proportion‚0.1005084291100502 +jet/accuracy_3_of_5‚0.08745519816875458 9"Purity/0FRt2SRqqt2FBt/SRqqt_purity‚0.5535714030265808 jet/accuracy_4_of_7‚0 9"Purity/*FRt3SRqqt2FBt/SRqqt_purity‚0.6666666865348816 ("Purity/1FRt3SRqqt2FBt/event_purity‚0 :"Purity/*FRt*SRqqt0FBt/SRqqt_purity‚0.49541547894477844 7 Purity/*FRt1SRqqt0FBt/FRt_purity‚0.1453859657049179 ,&Purity/0FRt3SRqqt2FBt/event_proportion‚0 9"Purity/2FRt1SRqqt*FBt/SRqqt_purity‚0.5458653569221497 8 Purity/3FRt*SRqqt*FBt/FRt_purity‚0.18333330750465393 ,&Purity/4FRt4SRqqt4FBt/event_proportion‚0 & Purity/3FRt0SRqqt1FBt/FBt_purity‚1 ,&Purity/*FRt2SRqqt4FBt/event_proportion‚0 7 Purity/3FRt2SRqqt0FBt/FRt_purity‚0.0833333358168602 & Purity/1FRt1SRqqt4FBt/FRt_purity‚0 ("Purity/2FRt2SRqqt2FBt/event_purity‚0 7 Purity/*FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 8 Purity/1FRt1SRqqt0FBt/FRt_purity‚0.14820219576358795 9"Purity/2FRt*SRqqt*FBt/SRqqt_purity‚0.5226680636405945 ,&Purity/2FRt4SRqqt4FBt/event_proportion‚0 +"Purity/0FRt0SRqqt4FBt/event_purity‚0.25 jet/accuracy_1_of_6‚1 7 Purity/*FRt2SRqqt2FBt/FBt_purity‚0.9633333683013916 & Purity/4FRt0SRqqt*FBt/FRt_purity‚0 ,&Purity/3FRt3SRqqt1FBt/event_proportion‚0 9"Purity/*FRt*SRqqt*FBt/event_purity‚0.2538914680480957 =&Purity/*FRt2SRqqt0FBt/event_proportion‚0.1085689514875412 :"Purity/0FRt0SRqqt2FBt/event_purity‚0.36079123616218567 =&Purity/1FRt0SRqqt0FBt/event_proportion‚0.1308283656835556 ?&Purity/2FRt1SRqqt1FBt/event_proportion‚0.007998512126505375 ,&Purity/2FRt1SRqqt4FBt/event_proportion‚0 9"Purity/3FRt*SRqqt0FBt/SRqqt_purity‚0.4746280014514923 >&Purity/*FRt3SRqqt*FBt/event_proportion‚0.02021329291164875 5&Purity/2FRt2SRqqt1FBt/event_proportion‚ 0.00390625 +jet/accuracy_3_of_4‚0.07963959127664566 ("Purity/0FRt3SRqqt*FBt/event_purity‚0 + Purity/3FRt3SRqqt0FBt/FRt_purity‚0.1875 & Purity/*FRt3SRqqt2FBt/FBt_purity‚1 *"Purity/0FRt2SRqqt3FBt/SRqqt_purity‚0.5 ("Purity/4FRt0SRqqt0FBt/event_purity‚0 9"Purity/2FRt2SRqqt0FBt/SRqqt_purity‚0.5167264938354492 7 Purity/3FRt4SRqqt0FBt/FRt_purity‚0.3333333432674408 ("Purity/4FRt*SRqqt*FBt/SRqqt_purity‚1 9"Purity/*FRt*SRqqt1FBt/SRqqt_purity‚0.5654616355895996 & Purity/*FRt4SRqqt*FBt/FBt_purity‚1 ("Purity/2FRt3SRqqt1FBt/event_purity‚0 ("Purity/3FRt3SRqqt0FBt/event_purity‚0 ("Purity/4FRt1SRqqt*FBt/event_purity‚0 ("Purity/1FRt3SRqqt*FBt/event_purity‚0 ("Purity/2FRt3SRqqt*FBt/event_purity‚0 8"Purity/3FRt2SRqqt*FBt/SRqqt_purity‚0.522988498210907 & Purity/4FRt2SRqqt0FBt/FRt_purity‚0 ;"Purity/*FRt1SRqqt2FBt/event_purity‚0.017786739394068718 8 Purity/*FRt2SRqqt1FBt/FRt_purity‚0.22927990555763245 @&Purity/0FRt1SRqqt3FBt/event_proportion‚0.0004340277810115367 particle/accuracy_1_of_6‚1 ("Purity/2FRt*SRqqt3FBt/event_purity‚0 ,&Purity/3FRt0SRqqt4FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt*FBt/event_proportion‚0 ,&Purity/4FRt3SRqqt4FBt/event_proportion‚0 8 Purity/*FRt3SRqqt1FBt/FRt_purity‚0.32499998807907104 9"Purity/3FRt*SRqqt1FBt/SRqqt_purity‚0.6666666865348816 * Purity/0FRt*SRqqt4FBt/FBt_purity‚0.875 7 Purity/*FRt*SRqqt0FBt/FRt_purity‚0.1562805473804474 @&Purity/2FRt4SRqqt0FBt/event_proportion‚0.0003100198518950492 ("Purity/3FRt0SRqqt2FBt/event_purity‚0 =&Purity/0FRt2SRqqt*FBt/event_proportion‚0.0535714291036129 & Purity/3FRt1SRqqt1FBt/FBt_purity‚1 A&Purity/*FRt*SRqqt4FBt/event_proportion‚0.00037202381645329297 9"Purity/*FRt2SRqqt*FBt/SRqqt_purity‚0.4918825924396515 8 Purity/1FRt*SRqqt*FBt/FRt_purity‚0.22371649742126465 9"Purity/1FRt1SRqqt1FBt/SRqqt_purity‚0.6064062714576721 ,&Purity/4FRt0SRqqt2FBt/event_proportion‚0 ,&Purity/4FRt1SRqqt1FBt/event_proportion‚0 & Purity/*FRt3SRqqt*FBt/FBt_purity‚1 7 Purity/*FRt2SRqqt*FBt/FBt_purity‚0.9845983386039734 @&Purity/*FRt4SRqqt*FBt/event_proportion‚0.001ǹ¬3020833721384406 9"Purity/1FRt*SRqqt0FBt/SRqqt_purity‚0.4943975508213043 ?&Purity/1FRt1SRqqt2FBt/event_proportion‚0.006138392724096775 @&Purity/3FRt2SRqqt1FBt/event_proportion‚0.0004340277810115367 ("Purity/4FRt*SRqqt0FBt/SRqqt_purity‚1 7 Purity/*FRt1SRqqt1FBt/FBt_purity‚0.9964737296104431 ("Purity/*FRt3SRqqt*FBt/event_purity‚0 =&Purity/2FRt0SRqqt*FBt/event_proportion‚0.0371403768658638 ,&Purity/3FRt3SRqqt4FBt/event_proportion‚0 particle/accuracy_3_of_5‚1 , Purity/1FRt3SRqqt1FBt/FRt_purity‚0.28125 8 Purity/3FRt0SRqqt0FBt/FRt_purity‚0.15909090638160706 9"Purity/1FRt4SRqqt*FBt/SRqqt_purity‚0.2916666567325592 A&Purity/1FRt1SRqqt3FBt/event_proportion‚0.00024801588733680546 :"Purity/1FRt2SRqqt*FBt/SRqqt_purity‚0.47760844230651855 *jet/accuracy_2_of_4‚0.3798464834690094 & Purity/3FRt2SRqqt1FBt/FBt_purity‚1 ("Purity/1FRt1SRqqt4FBt/SRqqt_purity‚1 & Purity/2FRt2SRqqt2FBt/FBt_purity‚1 ("Purity/0FRt2SRqqt3FBt/event_purity‚0 ;&Purity/*FRt*SRqqt0FBt/event_proportion‚0.61557537317276 :"Purity/1FRt*SRqqt2FBt/event_purity‚0.08674393594264984 A&Purity/4FRt*SRqqt*FBt/event_proportion‚0.00018601190822664648 ("Purity/3FRt4SRqqt0FBt/event_purity‚0 & Purity/0FRt0SRqqt1FBt/FBt_purity‚1 9"Purity/1FRt0SRqqt*FBt/event_purity‚0.2042236477136612 7 Purity/1FRt2SRqqt*FBt/FBt_purity‚0.9767857193946838 ("Purity/3FRt0SRqqt0FBt/event_purity‚0 7 Purity/0FRt2SRqqt3FBt/FBt_purity‚0.6666666865348816 >&Purity/*FRt1SRqqt2FBt/event_proportion‚0.01698908768594265 A&Purity/0FRt*SRqqt4FBt/event_proportion‚0.00024801588733680546 ,&Purity/1FRt2SRqqt4FBt/event_proportion‚0 ("Purity/3FRt0SRqqt*FBt/event_purity‚0 7 Purity/0FRt2SRqqt2FBt/FBt_purity‚0.9464285969734192 * Purity/4FRt*SRqqt*FBt/FRt_purity‚0.125 *jet/accuracy_1_of_4‚0.9126855134963989 & Purity/1FRt3SRqqt*FBt/FBt_purity‚1 & Purity/*FRt*SRqqt4FBt/FRt_purity‚0 9"Purity/*FRt1SRqqt0FBt/SRqqt_purity‚0.5218731760978699 7 Purity/1FRt1SRqqt1FBt/FBt_purity‚0.9980158805847168 7 Purity/1FRt2SRqqt1FBt/FRt_purity‚0.1982758492231369 A&Purity/1FRt4SRqqt*FBt/event_proportion‚0.00037202381645329297 :"Purity/2FRt*SRqqt2FBt/event_purity‚0.03488372266292572áY"CR‹+ 4xyaxc7dj@ ûÈãÔð‡¿0`Trainer.fit` stopped: `max_epochs=10` reached. í›fŸŒ+ 4xyaxc7djŒ üÈãÔìà­zEpoch 9/9 â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â”â” 1209/1209 0:14:20 • 0:00:00 1.41it/s v_num: xc7d)Bù%+ 4xyaxc7dj üÈãÔ€ßô­ Qàà2Ž+- _wandbruntime‚9486  _runtime‚9486q{Á+‚(’ŽJ